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Executive Summary 


Low- and middle-income countries have made significant progress toward placing children into 
schools, but student learning and achievement are often dreadfully low (Berry, Barnett, & 
Hinton, 2015; Pritchett, 2013). Globally, approximately 387 million children of primary school 
age are not acquiring basic reading and mathematics skills, even though about two-thirds are in 
school and will complete primary school (United Nations Educational, Scientific, and Cultural 
Organisation [UNESCO], 2017). 


Zambia faces many common educational challenges. Literacy rates among young Zambian 
adults ages 15-24 are 58.5% for females and 70.3% for males, despite an average of 7.7 years 
and 7.9 years of education, respectively (Central Statistical Office, Ministry of Health, & ICF 
International, 2014; UNICEF, 2015). Furthermore, public spending on education is low relative 
to other regional countries: Zambia spends about 1.3% of its gross domestic product (GDP) on 
education as compared with average expenditures of 5.6% of GDP in other southern and 
eastern African countries (UNICEF, 2015). Zambia also has a large, autonomous community 
schooling system that formed during the nation’s transition from a socialist economy. The 
community schooling system has expanded during the past 20 years to increase education 
access in remote areas. Estimates suggest that the number of Zambian community schools has 
increased from 100 schools in 1996 to about 2,325 schools with 473,458 children in 2017 
(Chimese, 2014; DeStefano, 2006; Ministry of General Education, Republic of Zambia [MoGE], 
2016). However, many community schools in Zambia are staffed by untrained, underpaid 
teachers who teach a substandard curriculum and who lack management skills and school 
supplies. These schools need a cost-effective solution for delivering quality education in order 
to improve learning outcomes. 


The eSchool 360 model implemented by Impact Network (IN) represents a promising approach 
to improving educational outcomes by incorporating three potentially high-impact components 
that could create important synergies: a standardized e-learning curriculum and pedagogy, 
ongoing teacher training and professional development, and community ownership. Each 
component could, on its own, have positive impacts on student outcomes by engaging the 
three main factors in the education system—students, teachers, and parents. Combining these 
components into a single program may be particularly effective by aligning all three factors 
toward improving the educational outcomes of students. Earlier research has suggested that 
these complementarities may be substantial, with even higher impacts from educational 
technology programs that include a strong focus on pedagogical practices (Muralidharan, Singh, 
& Ganimian, 2019). 
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The American Institutes for Research (AIR) has designed and is implementing a mixed-methods 
cluster-randomized controlled trial (cluster-RCT) to determine the effects of IN’s eSchool 360 
model. The study comprises three main evaluation components: an impact evaluation of the 
eSchool 360 model, an analysis of the cost-effectiveness of the eSchool 360 model, and a 
process evaluation of the expansion of the eSchool 360 model. To determine the impact of the 
program, we are using a cluster-RCT in which 63 eligible schools have been randomly assigned 
either to receive IN’s eSchool 360 program (30 treatment schools) or not to receive the 
program (33 control schools). The population consists of children ages 6—9 who live in close 
proximity to the 63 selected community schools across three districts—Petauke, Sinda, and 
Katete—in Zambia’s Eastern Province. The primary cognitive skills outcomes are aggregate 
scores on the early grade reading assessment (EGRA), early grade mathematics assessment 
(EGMA), the Zambian Achievement Test (ZAT), and Oral Vocabulary assessments. 


This report presents the midline results of the cluster-RCT used to determine the impact of IN’s 
eSchool 360 model. It examines the impact of the program on EGRA, EGMA, ZAT, and Oral 
Vocabulary assessments 14 months after the start of the program. In addition, it analyzes the 
impact of the program on intermediate outcomes, including school attendance and enrollment, 
perceptions of school and education quality, aspirations about child’s education and marriage, 
child development, food security, and education expenditures. We present two main types of 
effects. The first, called the Intention-to-Treat (ITT) effect, considers the impact on all children 
in close proximity of the schools, regardless of whether they attended the school. The ITT 
identifies the impact of the opportunity to attend an IN school. The second, called the 
Treatment Effect on the Treated (TOT), estimates the effect for those children who attend an IN 
school when given that opportunity. 


The estimated ITT effects suggest that the opportunity to attend an IN school positively 
affected children’s learning outcomes across the board. On average, we found statistically 
significant effects of 0.40 standard deviations or 3.5 percentage points on EGRA scores, 0.22 
standard deviations or 4.9 percentage points on EGMA scores, 0.16 standard deviations or 3.1 
percentage points on ZAT scores, and 0.25 standard deviations or 6.0 percentage points on Oral 
Vocabulary scores 14 months after the start of the program. These effect sizes are equivalent to 
1.88 additional years of education for reading and 1.03 additional years of education for 
mathematics if we extrapolate results of learning gains from Grade 1 to Grade 12 froma 
sample of representative low- and middle-income countries (Evans & Yuan, 2019). When 
compared directly against the one-year learning gains in the control group (who took the same 
tests and were measured on the same scale), the ITT estimates are equivalent to 1.2 additional 
years of education for reading and 0.37 additional years (or 4.5 additional months) of education 
for mathematics. 
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The estimated TOT effects suggest that the treatment effects on the treated for students whose 
parents chose for them to attend an IN school after getting the opportunity were substantially 
larger than the ITT effects. On average, participating in the eSchool 360 model resulted in 
improvements of 0.32 standard deviations or 6.3 percentage points in ZAT scores, 0.83 
standard deviations or 7.2 percentage points in EGRA scores, 0.45 standard deviations or 10.1 
percentage points in EGMA scores, and 0.52 standard deviations or 12.4 percentage points in 
Oral Vocabulary scores for students who reported that they attended IN schools three times in 
the last week. For students who reported that they had ever enrolled in IN schools (over the 
past year), we found TOT effects of 0.26 standard deviations or 5.2 percentage points in ZAT 
scores, 0.68 standard deviations or 5.8 percentage points in EGRA scores, 0.37 standard 
deviations or 8.2 percentage points in EGMA scores, and 0.42 standard deviations or 10.1 
percentage points in Oral Vocabulary scores. 


The results indicate that improved quality of education and increased school enrollment and 
attendance contributed to the positive effects on EGRA, EGMA, ZAT, and Oral Vocabulary 
scores. Residing in an IN catchment area increased the likelihood of school enrollment by 7.9 
percentage points. We also found statistically significant effects of the eSchool 360 model on 
parents’ satisfaction with education and school attendance. In addition, we found strong and 
statistically significant correlations between parents’ satisfaction with education and school 
attendance with EGRA, EGMA, ZAT, as well as Oral Vocabulary scores. 


We found less evidence for the contribution of other potential mechanisms to improvements in 
EGRA, EGMA, ZAT, and Oral Vocabulary scores caused by the eSchool 360 model. We did not 
find statistically significant effects of the eSchool 360 model on child development outcomes, 
including cognitive, social-emotional, and motor skills. 


Qualitative respondents (students, teachers, parents, and program staff) also reported a high 
perceived quality of education at IN schools in interviews and focus group discussions. Their 
experiences suggest that IN schools provided a higher quality education than community 
schools and that teachers were present more regularly at IN schools than at government 
schools. They also attributed improvements in literacy outcomes to the IN schools. However, 
they did not report improvements in mathematics. 


The qualitative research also showed evidence for a high level of fidelity of implementation, 
which likely contributed to the positive effects. The findings suggest that teachers used the 
participatory pedagogical approaches on which they were trained; that they adhered to the 
curriculum prescribed in the curriculum map; and that they used technology (tablets and 
projectors) as recommended. 
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Despite the positive program effects, the average reading and mathematics scores for children 
residing in IN catchment areas remained relatively low. They scored an average of only 10.9% 
correct on EGRA assessments and 24.3% correct on EGMA assessments, while children in the 
control group scored an average of 7.6% and 21.0% correct on EGRA and EGMA assessments, 
respectively. 


The results of the longer-term impact and cost-effectiveness analysis will show evidence on 
whether and how the eSchool 360 model can be moved to scale. We will estimate cost data of 
the eSchool 360 model in the first half of 2020. In addition, we will estimate longer-term 
impacts following the endline survey three years after the start of the program. We will present 
these results in a future endline report. 
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Introduction 


There have been dramatic increases in educational attainment worldwide during the past 
several decades, but the quality of education and overall student learning demand substantial 
continued improvement (Berry et al., 2015; Pritchett, 2013; World Bank, 2018). Globally, 
approximately 387 million children of primary school age are not acquiring basic reading and 
mathematics skills, even though about two-thirds are in school and will complete primary 
school (UNESCO, 2017). The current set of global development goals—the Sustainable 
Development Goals—shifts the focus from educational attainment to education quality, with 
the goal to “ensure inclusive and equitable quality education and promote lifelong learning 
opportunities for all” (United Nations, 2017, p. 24; emphasis added). 


Zambia is emblematic of many low- and middle-income countries that face multiple educational 
challenges. First, overall education quality is low: literacy rates among young Zambian adults aged 
15-24 are 58.5% for females and 70.3% for males, despite an average of 7.7 years and 7.9 years 
of education, respectively (Central Statistical Office, Ministry of Health, & ICF International, 
2014; UNICEF, 2015). Second, Zambian public spending on education is low relative to other 
regional countries: Zambia spends about 1.3% of its GDP on education as compared with an 
average education expenditure of 5.6% of GDP in other southern and eastern African nations 
(UNICEF, 2015). Third, Zambia has a large, autonomous community schooling system that formed 
during the country’s transition from a socialist economy. The system has expanded over the past 
20 years to increase education access in remote areas. Estimates suggest that the number of 
community schools has increased from 100 schools in 1996 to about 2,325 schools with 473,458 
children in 2017 (Chimese, 2014; DeStefano, 2006; MoGE, 2016). However, community schools 
are often staffed by untrained, underpaid teachers who teach a substandard curriculum and who 
lack management skills and school supplies. Improving education quality in community schools 
may be an effective entry point to improve educational outcomes for vulnerable children in 
remote areas. 


This midline study focuses on the impact and fidelity of implementation of IN’s eSchool 360 
model 14 months after program start. The eSchool 360 model represents a potentially 
promising approach to delivering quality education and improving educational outcomes for 
students in community school in rural Zambia. It incorporates three potentially high-impact 
interventions that could offer important complementarities: a standardized e-learning 
curriculum and pedagogy, ongoing teacher training and professional development, and 
community ownership. The e-learning component provides electricity via solar power (supplied 
by IN), and projectors and tablets (supplied by IN’s partner, Mwabu) for the community 
schools. The tablets are loaded with local-language materials that are structured around a 


AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 5 


Midline Report for the Mixed-Methods Cluster-Randomized Controlled Trial of 
Impact Network’s eSchool 360 Model in Rural Zambia 


curriculum approved by the Zambian government. IN supplements the technology by providing 
teacher coaching, observing teacher practices in the classroom, and creating community 
ownership. Locally hired teachers receive weekly training focused on using the technology and 
enhancing their pedagogical skills. 


Combining e-learning, ongoing teacher training and professional development, and community 
ownership components into a single program may be particularly effective by aligning the 
incentives of students, teachers, and parents toward improving student educational outcomes. 
These components could each, on their own, have positive impacts on student outcomes. 
Earlier research has demonstrated that engaging all three factors—students, teachers, and 
parents—in the education system may be particularly effective because it could create 
important synergies. For example, an educational technology program in urban India that 
included a strong focus on pedagogical practices showed positive effects that were greater than 
the sum of those obtained from separate educational technology or pedagogical interventions 
(Muralidharan, Singh, & Ganimian, 2019). A cost-effectiveness analysis of the same program 
found that it also was cost-effective in improving learning outcomes. This is important 
considering that a recent review found that technology-based education programs may not be 
cost-effective, even if they produce large impacts on learning outcomes (Muralidharan et al., 
2019; Piper, Zuilkowski, Kwayumba, and Strigel, 2016). 


Joel impact school. 
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This study comprises three main evaluation parts: an impact evaluation of the eSchool 360 
model, an analysis of the model’s cost-effectiveness, and a process evaluation of the model’s 
expansion. This report presents midline findings related to the impact and process evaluation. 
In addition, we have produced an inception report that details the design of the cost- 
effectiveness analysis (De Hoop et al., 2017) and a baseline report that summarizes the 
quantitative data collection at baseline (De Hoop, Brudevold-Newman, & Davis, 2018). We will 
collect the cost data in 2020 and will present an analysis of these data in the endline report. 


Below we present research questions related to the impact evaluation, cost-effectiveness 
analysis, and process evaluation components of this study. Each part of the evaluation is 
designed to answer different but complementary questions. 


Impact Evaluation 


a. What is the effect of the eSchool 360 program on students’ numeracy, preliteracy, and 
literacy skills? 


b. Do students enrolled in the eSchool 360 program improve in numeracy and literacy 
skills? 


c. Does the eSchool 360 program increase attendance and enrollment? 


d. Does the eSchool 360 program lead to an improved perception of school and education 
quality among students, teachers, and parents? 


e. Does the eSchool 360 program improve parental and children’s aspirations? 


Cost-Effectiveness 
a. How cost-effective is the eSchool 360 program in improving literacy outcomes? 


b. How cost-effective is the eSchool 360 program in improving mathematics outcomes? 


Process Evaluation 


a. Was the eSchool 360 program implemented as designed? If not, why was it not 
implemented as designed, what were the challenges to implementing it as designed, 
and how was it implemented? 


b. How did the eSchool 360 program implementation vary by geography, culture, and time 
of year? 


c. Did perceptions of the quality of teachers differ among students, parents, teacher 
supervisors, and teachers? If yes, how? 
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The remainder of this midline report is structured as follows: The report begins with a 
description of the background of this study and includes an overview of the existing literature 
on the impact of technology-based education programs. Next, this report presents a description 
of the eSchool 360 model followed by a description of its theory of change and the research 
designs for the impact and process evaluations. The report then offers a detailed overview of 
the quantitative and qualitative midline results and discusses conclusions. 


Background and eSchool 360 Model 


The current Zambian educational system faces many challenges in providing quality education 
and remains inaccessible to many Zambians under the age of 18 (i.e., 52.5% of the nation’s 
population) (Central Statistics Office Zambia, 2013). An estimated 600,000 students attend 
nongovernmental, autonomous community schools that do not offer a full range of grades, are 
in poor condition, and are funded through minimal government funding of less than U.S. $91 
per year (1,000 Zambian Kwacha) per school (DeStefano, 2006). The Zambian government has 
introduced evidence-backed interventions in government schools to improve school quality, 
such as the Teaching at the Right Level program. This intervention groups students according to 
learning level rather than by age or grade (Banerjee et al., 2016). However, the autonomous 
nature of community schools raises concerns related to whether they may be neglected from 
this quality push as well as whether a different set of interventions may be necessary to 
improve their quality. 


Previous research demonstrates that multifaceted education programs such as IN’s eSchool 360 
model can improve learning outcomes. A comprehensive, systematic review on the impact of 
education programs in low- and middle-income countries concludes that successful education 
programs address constraints at multiple levels (Snilstveit et al., 2016), which can only be 
achieved by multifaceted education programs. Kremer, Brannen, and Glennerster (2013) also 
highlight the importance of adapting the curriculum to the child’s level to improve the 
effectiveness of pedagogical interventions. Furthermore, Conn (2014) suggests that among 
interventions that include teacher training as a component, those with adaptive instruction had 
larger effect sizes than those without adaptive instruction. Finally, Muralidharan et al. (2019) 
find that in an RCT of a personalized, computer-aided afterschool instruction program in India, 
students in the treatment group made significant gains in mathematics and Hindi test scores. 
The authors concluded that the impact was due primarily to the computer-aided learning 
system’s ability to target and adapt to wide variations in student learning levels. 


In contrast, there is evidence that programs in low- and middle-income countries that focus on 
increasing educational inputs without addressing other learning constraints are not sufficient to 
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improve learning outcomes (Schling & Winters, 2018; Snilstveit et al., 2016). An increased 
provision of traditional school resources such as textbooks or flipcharts had no impact on 
student attainment (Glewwe, 2002). Banerjee, Cole, Duflo, and Linden (2007) note that 
increasing inputs to schooling fails to have an impact on student attainment if what is being 
taught remains too difficult for students to learn. Similarly, numerous studies that have focused 
on computer-assisted learning programs did not find significant impacts when the program 
focused only on increased educational inputs (Snilstveit et al., 2016; Valk, Rashid, & Elder, 
2010). 


The eSchool 360 Program. IN developed the eSchool 360 model to deliver low-cost education 
to children in rural communities through a holistic solution. A previous evaluation of the 
eSchool 360 model showed that the cost of the program was $3 per month per student, which 
is 70% less than the Zambian government spends per student (Winters, Schling, & Winters, 
2013). The core of the eSchool 360 model is e-learning technology whereby tablets and 
projectors, provided by IN's partner, Mwabu, are loaded with both lesson plans for teachers 
and with interactive lessons for students, approved by the Zambian government, and delivered 
in the local language. IN supplies electricity via solar power and supplements the technology by 
providing teacher training and professional development and creating community ownership. 
Locally hired teachers receive weekly training focused on using the technology and enhancing 
teachers’ pedagogical skills. The approach represents a significant innovation not only because 
technology is used but also because it incorporates the practice of training local high school 
graduates to be teachers and provides them with systematic, ongoing support. 


Encouraging results of a previous nonexperimental but longitudinal evaluation of the eSchool 
360 model led AIR to fund the expansion of the eSchool 360 model to 30 additional community 
schools in rural Zambia (from an initial core of nine community schools, making for a total of 39 
community schools receiving the program).' The earlier analysis indicated that improving 
mathematics outcomes by 1 percentage point cost IN schools 88% less than it cost government 
schools, suggesting that the model may be cost-effective in improving learning outcomes in 
Zambia (Schling & Winters, 2018). IN conducted the expansion in 2017 by implementing the 
eSchool 360 model in 30 community schools across three rural Zambian districts (Katete, Sinda, 
and Petauke) in areas with no running water and limited electricity. The cohort of students that 
was studied for our evaluation was admitted into the program in January 2018. 


1 AIR decided to fund the program’s expansion to include five quasi-governmental schools that were not participants in the 
cluster-RCT. We will compare learning outcomes of the students in these five schools to the learning outcomes of students in 
five comparable governmental schools during the final data collection, which will be three years after the introduction of the 
eSchool 360 model in the quasi-governmental schools. 
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AIR and IN designed the expansion to include a rigorous mixed-methods cluster-RCT to determine 
the impact of the program on students’ learning outcomes. To achieve this goal, AIR and IN 
closely consulted with Zambian government officials to obtain letters of approval for random 
assignment of the eSchool 360 program to 30 treatment schools and 33 control schools.” The 
random assignment of schools was conducted in May 2017. The Zambian Ministry of Education 
officials implemented the randomization, while AIR staff ensured the integrity of the process. We 
chose an unbalanced design with a smaller number of treatment schools because of limited 
resources to implement the eSchool 360 model. 


Theory of Change 


The program’s theory of change suggests that the eSchool 360 program may lead to 
improvements in learning outcomes through various mechanisms (Figure 1). First, the teacher 
professional development component of the model may produce improvements in knowledge 
and practices of untrained teachers, which may result in improvements in the quality of 
education—for example, through the integration of activity-based learning methods and 
improvements in the curriculum. These improvements, in turn, may lead to improvements in 
preliteracy, early grade reading, and early grade mathematics outcomes. Second, the 
infrastructure improvements in the community school may lead to increased demand for 
education, which may result in increased education enrollment and attendance. The 
infrastructure improvements may also result in decreases in the age-at-enrollment of Zambian 
students. These improvements in school attendance and enrollment may then result in an 
increase in the time spent on education, which may lead to improvements in learning 
outcomes. Third, the incorporation of technology may enable untrained teachers to be more 
effective because pre-prepared lessons are sequenced and built into the tablets, which are 
regularly checked and monitored by operations management. 


In addition to the improved learning outcomes, the program may result in improvements in the 
aspirations of students and parents. Improvements in the quality of education may increase 
expectations for students’ futures. These increased expectations may, in turn, lead to higher 


? An earlier version of this report included 34 control schools and 30 treated schools. However, discussions with program 
implementers from Impact Network showed that researchers had mistakenly included an excluded school to the treatment 
schools and a treatment school to the control group in the baseline analysis and the earlier report. Impact Network had initially 
identified 64 eligible schools (as opposed to 65 schools as reported in the inception and baseline report). However, further 
analysis suggested that the 64" school was too far away from the Impact Network offices to be considered for the program. 
This school had initially been assigned to the treatment group, but we replaced the school with a randomly selected school 
from the control group in the same district. This change was initially not taken into consideration in the analysis and baseline 
and midline data were collected in the catchment areas of this school as well. The current report presents results after 
correcting the sample composition (we removed the ineligible school from the analysis and switched the control group school 
to the treatment group for which it was earlier selected), and therefore includes slightly different sample sizes and impact 
estimates in comparison with the previous report. 
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aspirations in the domains of education, labor market, and family outcomes. For example, 
parents may increase their expectations of the likelihood that their children will be able to 
finish school. In addition, the improved quality of education may result in increased 
expectations of the returns of education, which may lead to higher expectations for labor 
market outcomes. Finally, increased aspirations in the education and labor market domains 
may result in increases in expectations relative to students’ marriage prospects and increased 
age at marriage. 


The validity of eSchool 360’s theory of change depends on several assumptions. Perhaps most 
important, teachers need to comply with eSchool 360 programming. In addition, the 
community schools must have sufficient capacity to implement the model. Furthermore, locally 
selected teachers need to have appropriate incentives to provide quality education. The 
language of instruction also must be consistent with the needs of the student population. 


The effects of the model may also vary with several individual-, household-, and community- 
level moderators. For example, the effects may vary by gender, age, and socioeconomic 
household-level characteristics. In addition, the model may be less effective in improving school 
attendance and enrollment for students who live farther away from the IN school. Finally, the 
model’s impacts may be moderated by student baseline preliteracy, reading, and mathematics 
outcomes, as well as parental education levels. We test each of these potential heterogeneities 
in the impact evaluation. 
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Figure 1. Theory of Change 


* Public spending on education 
constitutes only 1.3% of gross 
domestic product as compared 
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southern and eastern Africa. 


* Literacy rates among young 
adults ages 15-24 are only 
58.5% for females and 70.3% 
for males (UNICEF, 2014) 

* Ofthe 14 sub-Saharan countries 
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and Eastern Africa Consortium 
for Monitoring Educational 
Quality, Zambian students 
attained the lowest marks in 
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(SACMEQ III, 2010-2012) 

* The Zambian government funds 
community schools with less 
than $91 per year per school 

* The number of community 
schools in Zambia has grown 
from 100 in 1996 to 3,000 with 
600,000 children today 
(Chimese, 2014) 

* Community schools are often 
staffed by untrained and 
underpaid teachers; they lack 
management and school 
supplies, and they teach a 
substandard curriculum. 

* Many community schools have 
fallen into disrepair and do not 
offer the full range of primary 
grades 
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Impact Network eSchool 360 Theory of Change 


Community Outreach 
* Community is aware of Impact Network program 
* Students are able to attend Impact Network schools 


School Renovation 
* Improved school infrastructure 


Teacher labor conditions 

* Higher teacher salaries 

* Improved secondary labor conditions 
* Contributions to retirement fund 


Price of Education 
* Free access to education 
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* Perceived quality of education increases 
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* Increased school enrollment 
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* Lower age at enrollment 


Engagement with Ministry of Education 
* Obtain buy-in from Ministry of Education 
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Curriculum 

* Improved activity-based 
curriculum 

* Increased relevance of skill 
sequence and scope 

* Increased cultural and gender 
sensitivity and appropriateness 


Teacher professional development 

* Teacher professional 
development on weekly basis 
(coaching) 

* Monthly teacher training. crafted 
by teacher supervisors 


Parent-Teacher Association 
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Educational Materials/Technology 

* Introduction of tablets 
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Activity-based classroom practices 
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Technology-based education 
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Teacher knowledge of pedagogical 
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technology 
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Child engagement 

+ Increased child enrollment in 
school at a younger age 

* Increased child attendance in 
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Community support 

* More support from parents 
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Teacher practices 

* Increased time spent in school 
and “learning time” 

+ Improved teacher pedagogical 
practices 


Student level learning outcomes 
* Increased pre-literacy literacy 
skills 
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siblings, # children, household size) 


Teacher compliance with e-school programming 
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Language of instruction is consistent with student population’s needs 


Distance to school 
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Study Design and Methods 


AIR has designed and is implementing a mixed-methods cluster-RCT. The data collection began 
with quantitative baseline data, followed by quantitative and qualitative data collection 14 
months after the start of the baseline data collection. We will collect endline data three years 
after program start. In addition to the quantitative and qualitative impact analyses, we will 
conduct a cost-effectiveness analysis. Specifically, we will assess the costs of the eSchool 360 
model using the ingredients method. For this purpose, we will need to specify all ingredients 
that are necessary to replicate the model and then collect data on the unit costs of these 
ingredients (Dhaliwal, Duflo, Glennerster, & Tulloch, 2011). AIR will work with IN to gather 
information on resources used for the intervention to create an exhaustive list of resources 
with costs. Using this information, AIR will create a cost database. We will then estimate the 
costs of the intervention for the average beneficiary and divide these costs by the expected 
gain in outcomes derived from the impact analysis to serve as the cost-effectiveness measure 
of the intervention. 


This report presents findings from the midline study, which comprised quantitative and 
qualitative data collection. The inception report includes a description of the methods we 
propose to use for the cost-effectiveness analyses (De Hoop et al., 2017). 


Impact Evaluation Design 
Sampling and Randomization 


The cluster-RCT evaluation of the eSchool 360 model involved randomly assigning the program 
among schools that satisfied IN’s geographic, infrastructure, and organizational structure 
eligibility criteria for the eSchool 360 expansion. The geographic criteria arose from IN’s goal of 
introducing community schools across three districts—Petauke, Sinda, and Katete—in Zambia’s 
Eastern Province. Of the schools in these districts, IN sought those with a dedicated physical 
structure, that were largely informal, and that had more community teachers than government 
teachers. Pairs of eligible schools that operated within three kilometers of one another were 
excluded to reduce bias from spillovers or contamination, which may occur if a child intended 
to be unaffected by the eSchool 360 model subsequently attends an IN school. 


IN and AIR first consulted with local Zambian government officials to obtain a list of all 
community schools in the region. Of the 149 community schools that were identified, 63 met all 
eligibility criteria. IN staff then visited each of the 63 schools to obtain information on the 
structure of the school, the number of government and volunteer teachers, the state of the 
infrastructure, the grades served, and distance to other schools. Local representatives of the 
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Ministry of Education implemented the randomization with oversight from AIR, assigning five 
schools in Katete, nine in Sinda, and 16 in Petauke to the treatment group (the group receiving 
the eSchool 360 model). The remaining 33 schools were assigned to the control group (the 
group not receiving the model). 


This study estimates the impact of the eSchool 360 model on children who are eligible to enroll 
in first grade and who live near one of the 63 schools. We focused on these children because 
the introduction of the eSchool 360 model was designed to expand to an additional grade each 
year of the model’s operation. In the first year, only the first-grade cohort received the full 
program package. This will expand to Grades 1 and 2 in the second year, and so on. The study 
will use a longitudinal panel design that follows each of the sampled children for three years, 
regardless of where and when they enroll in school, to estimate (1) Intention-to-Treat, or ITT 
effects of the model, which identifies the effect of the opportunity to enroll in an IN school; and 
(2) Treatment-Effects-on-the-Treated, or TOT effects, which identify the effect of attending an 
IN school on those who actively participate in the program (i.e., attend IN schools). 


For the ITT analysis, we compared all eligible children who lived near the 30 IN schools with 
eligible children who lived near the 33 control schools, thus including all children regardless of 
whether they enrolled in a school. To identify the sample of children with the potential to be 
affected by the model and to then determine ITT effects, we conducted a census in the areas 
surrounding the sample schools to identify all households with children eligible to enroll in first 
grade in January 2018; that is, children ages 6 years or older in January 2018 who did not attend 
first grade or higher in the prior school year. We identified all households with children eligible 
to enroll in first grade within a diameter of 1.5 kilometers of the schools. We iteratively 
expanded the distance by 0.5 kilometers in communities with insufficient numbers of eligible 
children within the initial or subsequent sampling areas until we found sufficient eligible 
households. This procedure was implemented consistently across treatment and control 
school-catchment areas.? 


3 The initial distance stemmed from conversations with local experts who suggested that children eligible for first grade 
generally do not walk more than 1.5 kilometers to school. Since those conversations were conducted, we have identified a 
number of cases in which children walk longer than 3 hours to attend school. 
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Figure 2. Map of Sample Schools 


Mkumpa 


Katete 


It is important to identify ITT effects because the announcement of the treatment schools and the 
construction of the additional infrastructure involved with the implementation of the eSchool 360 
model are visible to the beneficiaries. The randomization of schools to treatment occurred nearly 
seven months before the first evaluation cohort was admitted to the sample schools in January 
2018. This visibility may result in a different composition of students attending the treatment 
schools relative to the control schools—for example, by influencing school enrollment and 
attendance. Such changes in the composition could result in a bias in the impact estimator that 
compares students enrolled in the IN schools with students enrolled in other community schools 
because of differences in either observable (e.g., age, gender, parents’ education level) or 
unobservable (e.g., motivation, noncognitive skills) characteristics. 


To address concerns regarding these composition effects and to enable the estimation of ITT 
effects, we randomly sampled 30 households from the census-generated sample frame for each 
of the sample schools. For households with more than one eligible child, we selected the oldest 
child for inclusion in the sample. The introduction of IN schools may have induced some eligible 
children to enroll in school (who would not have enrolled in school absent IN). Comparing the 
outcomes (e.g., test scores) of all eligible children allows for capturing any effects on inducing 
children to attend school. 
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To increase statistical power, we oversampled 8-year-olds and 9-year-olds from our sample 
based on descriptive statistics about the age distribution from the Eastern Province (Ministry of 
General Education, 2014). These descriptive statistics indicated that 8-year-olds and 9-year-olds 
are more likely than 7-year-olds and 6-year-olds to be enrolled in first grade (Ministry of 
General Education, 2014). For our sampling strategy, we began by randomly selecting 
households with 7-year-olds, 8-year-olds, and 9-year-olds within a distance of 1.5 kilometers 
from each of the schools. However, we had to slightly adjust the sampling strategy because 42 
catchment areas did not have sufficient numbers of 7- to 9-year-old children.* The sample was 
adjusted by randomly sampling additional 6-year-old children, followed by sequentially 
expanding the radius of the circle by an additional 0.5 kilometers until a sufficient sample was 
identified. Figure 3 presents the consort flow chart that describes the sampling strategy, starting 
with the eligibility criteria, followed by the random assignment, the census, the planned 
stratification by age, and the actual stratification following the practical decisions that we made 
during data collection. 


4 A “catchment area” is defined as the geographic area served by a particular school. 
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Figure 3. Consort Flow Chart 
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Respondents by age: 
6-year olds (nm = 112) 
7-year olds (n = 352) 
8-year olds (n = 215) 
9-year olds (n = 209) 
Total (n = 888) 


Respondents by age: 
6-year olds (n = 109) 
7-year olds (n = 351) 
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Data collection: 42 school catchment 
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have enough7-9-year olds available at 

time of visit, sosome 
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Quantitative Data Collection 

AIR is partnering with Palm Associates, a Zambian research organization that specializes in data 
collection and social science research, to collect data for the evaluation of IN’s eSchool 360 
model. AIR worked closely with Palm to train enumerators before the baseline and midline data 
collection and then followed enumerators into the field to observe data collection. Palm works 
with Zambian enumerators who speak the local language of the areas included in the study and 
who are familiar with the assessments that AIR will implement. 
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Palm used tablets to conduct the baseline data collection, thus improving the quality of data 
collection, minimizing the need to clean data, and eliminating data entry. We collected the data on 
tablets running SurveyCTO software, which minimizes errors in the field because skipping patterns 
can be automated and built-in checks ensure the quality of data. In addition, the tablets allow for 
the collection of GPS data, which enables the tracking of households from baseline to midline data 
collection in the field. 


The primary achievement indicators for numeracy, preliteracy, and literacy come from the EGRA, 
EGMA, and ZAT, which have already been adapted and used in the same region as this study. In 
addition, we collected Oral Vocabulary assessment data on the students. These instruments have 
been translated and validated in the context of Zambia. We collected assessment data primarily in 
the Nyanja and Chewa dialects because these are the languages of instruction in Grades 1-3 in 
Zambia’s Eastern Province. Further, we collected enrollment and attendance data during the 
midline household survey as well as outcome indicators on parental and community perceptions of 
schools, teachers, and their children’s education. Finally, we collected outcome data on parents’ 
and children’s educational, labor market, and marriage aspirations. In addition to these outcome 
indicators, the survey incorporated other moderating variables at different levels, including (1) 
student-level—gender and age; and (2) household-level—distance from school, poverty level, 
parents’ education level, and household size. During the endline data collection, we also will 
collect data at the school level, including data on school size, number of teachers, number of 
teaching years, teacher age, and average class size. 


The baseline report included a detailed overview of the construction of outcome variables (De 
Hoop et al., 2018). 


Cluster-RCT Analysis 

The main impact analysis used an ITT analysis to measure the impact of living within 1.5 
kilometers (or within three kilometers for a small minority of the children) of an IN school on 
student academic outcomes, school attendance, and perceptions of the quality of education, 
regardless of whether the student chose to attend the IN school. We used an ANCOVA model to 
estimate the ITT effect 14 months after the start of the program. The ANCOVA approach uses a 
regression specification that includes the baseline measures of outcome variables as an 
additional explanatory variable. 


The proposed evaluation design, with random assignment of schools to treatment, provides an 
unconfounded measure of the direct effect of the IN intervention on student outcomes. The 
probability of assignment to treatment is orthogonal to individual characteristics after 
controlling for stratum fixed effects. Thus, the direct effect of treatment (residing in an IN 
school catchment area) on outcome Y; can be estimated using the regression specification: 
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Yi2019 =A+ BIN, + 6S; + OYizo1g + UG + & 


Here, IN; is an indicator variable equal to 1 if individual i resides in the catchment area of an IN 
school and equal to 0 otherwise, S; is a vector of dummy variables for the district strata, Yj291 is 
a baseline value of the outcome of interest, C; is a vector of other control variables, and €; is a 
conditionally mean-zero error term. Since treatment was randomized within strata, the inclusion 
of the Y;291g, S;, and C; variables should increase efficiency but not impact the estimated value 

of £. If, by chance, there is some imbalance in the baseline covariates between the treatment and 
control groups, the regression model adjusts the outcome values to account for these 
differences, which leads to a more precise estimate of 6. We used cluster-robust standard errors 
clustered at the school level to account for potential correlation in outcomes within a school 
catchment area. To bring the sample to being representative of the population in terms of 
likelihood of enrolling in school by age, we applied post-stratification weights to our analyses. We 
report all our impact estimates in two units — the original score unit measured in terms of 
percentage-points (as a score out of 100), and standardized scores where each student’s midline 
score is standardized to the midline mean and standard deviation of the control group.° 


To address the potential inflation of Type | error and statistical significance owing to multiple 
comparisons, we applied corrections for multiple comparisons to multiple outcome measures 
within the same outcome domain using the Benjamini-Hochberg procedure, as recommended 
by the What Works Clearinghouse and employed by Banerjee et al. (2015). The outcome 
measures for the impact analyses are organized into two domains: primary EGRA outcomes and 
primary EGMA outcomes. These outcomes are single measures and are not corrected for 
multiple comparisons. The EGRA and EGMA subtask outcomes are related within the domains, 
and the results are corrected for multiple comparisons. We also applied corrections for multiple 
comparisons in the estimation of heterogeneous effects. Finally, we conducted a robustness 
check by applying randomization inference, which examines whether the results are robust to 
any variations in the data that may have arisen from the randomization by calculating the 
proportion of alternative realizations of the regression coefficient that are further away from 
zero than the estimated regression coefficient (Hef, 2017; Young, 2019). 


> We chose to standardize against the midline score of the control group because we consider the midline score of 
the control group more relevant and more representative for first-grade children in community schools in Zambia 
than the baseline score for the full sample. Some other impact evaluations of education programs in international 
development chose to standardize against the baseline sample mean and standard deviation (see for example, 
Muralidharan et al., 2019). However, in our study, none of the children attend school at baseline, leading to little 
variation across the sample. Using the baseline sample mean and standard deviation for standardization would 
thus result in inflated effect sizes when we would communicate effect sizes in standardized mean differences. 
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We estimated treatment effects on the treated, which reflect the effects of attending an IN 
school, by using the program assignment as an instrumental variable for self-reported school 
attendance in IN schools. The TOT is the impact of the intervention on those children who 
participated in the eSchool 360 model. Within this group we distinguish between children who 
actively participated in the program (children who self-reported attending IN schools at least 
three times in the week before the survey), and children who had ever enrolled in IN schools in 
the last year (children who may have participated in the eSchool 360 model but likely less so 
than the subgroup of children who attended IN schools at least three times in the week before 
the survey). 


The theory of change identified a number of potential moderators to the impact of the IN 
model. To account for these moderators, we tested for whether the program had a differential 
impact by age; gender; region; socioeconomic status; mother’s education level; and baseline 
EGRA, EGMA, and ZAT attainment. For noncategorical moderators, we constructed a binary 
variable equal to 1.0 for individuals with a value greater than the median and equal to 0.0 for 
those with a value less than the median. We then interacted the binary variable with treatment 
to test for whether the treatment was statistically different for groups with high and low levels. 
For socioeconomic status, we constructed an asset index using the calculated values from the 
first principal component of a list of assets as recommended by Filmer and Pritchett (2001). 


Estimating Equivalent Years of Schooling 

Finally, we transformed the effects on EGRA and EGMA outcomes to equivalent years of 
schooling using two different methods. For the first method, we used estimates from Evans and 
Yuan (2019), who show that students learn between 0.15 and 0.21 standard deviation of 
literacy ability over the course of a business-as-usual school year in a sample of representative 
low- and middle-income countries. Specifically, we use the estimates from their regression 
model for the pooled sample of Bolivia, Colombia, Ghana, Kenya, and Vietnam. The model 
explains reading proficiency with the number of years of schooling and a number of control 
variables and estimates the learning gain from an additional year of schooling based on the 
learning gain from Grade 6, usually the last year of primary school. We used the regression 
model that does not include country fixed effects, which suggests that 4.7 years of schooling 
results in a one standard deviation improvement in reading proficiency.®° For the second 
approach we used estimates of the learning gains in the original test scale for the control group 
since the baseline survey. These learning gains are a weighted average of learning gains for 
children who do and who do not attend primary school in the year before the survey. This 


5 Evans and Yuan (2019) standardize scores against the mean and standard deviation of the population scores, and 
therefore the comparison between our effect sizes and results from their study may not be perfectly reconcilable. 
Nevertheless, we report the results to enable a comparison with other education programs in international 
development settings. 
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comparison may be more valid for interpreting learning gains against a counterfactual setting, 
since all children in the control group are tested on the same assessment and are scored on the 
exact same scale as the treatment group. 


Process Evaluation Study Design 

AIR is also implementing a process evaluation to assess the fidelity of implementation of the 
eSchool 360 program. The process evaluation complements the impact evaluation by enabling 
us to explore informal patterns and unanticipated interactions in addition to formal activities 
and anticipated outcomes, while also giving us flexibility to explore unforeseen areas of 
interest. Incorporating a qualitative process evaluation into a larger impact evaluation allows us 
to investigate why we do or do not see impacts and to explore nuances and variations in our 
sample, particularly with regard to sensitive or complex topics. Further, this component of the 
study provides rich contextual information to complement the numerical findings of the impact 
evaluation while also allowing us to triangulate and explain quantitative findings. 


Qualitative Data Collection 

We used three primary approaches to collect qualitative data for this evaluation: key informant 
interviews (KIls) with teachers, teacher supervisors, and IN program staff; focus group 
discussions (FGDs) with students and parents; and classroom observations. 


KIls—Klls with carefully selected teachers, teacher supervisors, and IN staff are an important 
element of our evaluation design because they shed light on perceptions of the quality of the 
eSchool 360 model both in terms of how the technology and teacher trainings have affected 
classroom instruction and how the program engages with school communities. Throughout Kils, 
respondents discussed the perceived impact of the program, and we explored respondent 
beliefs about whether the program is being implemented as planned and students are being 
instructed as intended. 


FGDs—FGDs provide a context in which participants feel comfortable and empowered to 
discuss the evaluation topics with their peers and the carefully trained facilitators. We created a 
social dynamic that encourages participants to reflect upon their opinions and experiences and 
express them verbally. FGDs provided an opportunity to gain a deeper understanding of the 
factors enabling or inhibiting implementation of the eSchool 360 curriculum in the classroom. 
From FGDs, we investigated experiences that parents and PTA members had when 
communicating with IN staff, generally engaging with the program, and using the school in the 
evenings and on weekends. With students, we discussed the techniques that teachers used in 
the classroom as well as the friendliness of the school environment. 


Classroom Observations—We conducted formal classroom observations at one treatment 
school in each of the three districts. Classroom observations enabled us to examine the extent 
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to which teachers at IN schools were effectively using e-learning technology and applying 
pedagogical techniques gained during eSchool 360 teacher trainings. Observations also noted 
the eSchool 360 environment, particularly with respect to its child-friendliness. We conducted 
classroom observations in one randomly selected academic class at each of the three selected 
schools using a piloted and validated instrument. 


Klls, FGDs, and classroom observations took place in one treatment school in each of the three 
districts of Katete, Petauke, and Sinda. In each district, the research team worked with IN staff 
to select one school to visit. The selection was made based on a list of observable 
characteristics including school size, distance from district center, and the distribution of 
high/low performing schools based on student learning outcomes from prior years. Table 1 
provides a summary of the qualitative data collection approaches and respondents. 


Table 1. Summary of Qualitative Methods for eSchool 360 Process Evaluation 


Method Respondents Number 


Classroom observation N/A 


AIR partnered with Palm Associates to conduct qualitative data collection. Two experienced 


enumerators were recruited by Palm Associates to carry out interviews, focus groups, and 
classroom observations. Researchers spoke the Nyanja and Nsenga dialects, and qualitative 
instruments were translated into these languages during a three-day training. The researchers 
digitally recorded all KIls and FGDs on portable digital recorders. They downloaded recordings 
to field laptops each day, renamed according to an anonymized code system held in an 
encrypted spreadsheet and copied to external media for backup. At the end of each day, the 
field researchers transcribed the handwritten field recording sheets to word processed 
documents, translating the material as necessary. 


Researchers used audio recordings to supplement and validate the written transcriptions and 
translations. They assigned all transcriptions new names according to the code system in order 
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to ensure data and informant confidentiality. The data collection supervisor ensured that the 
work was carried out in accordance with research plans, on schedule, ethically, safely, and 
completely. The supervisor also reviewed transcribed interviews to ensure high standards of 
data quality. 


Qualitative Data Analysis 

All data from Klls and FGDs were coded and analyzed using the NVivo qualitative software 
program. Our team created a preliminary coding outline and structure based on the research 
questions, interview protocols, and memos of ideas that emerged during data collection. This 
coding outline served as a tool to organize and subsequently analyze information gathered in 
the interviews and focus groups. We created a list of definitions for the codes to ensure that 
coders categorized data using the same standards. After inputting the raw data into NVivo, 
coders selected a sample of interviews to double-code in order to ensure interrater reliability. 
Using these coded data, the qualitative team applied grounded theory to identify themes, 
categories, and theories that emerged from the data and that confirmed or refuted the 
researchers’ initial impressions. In other words, rather than basing the analysis on a hypothesis, 
the researchers created concepts and categories based on the data, refining the concepts 
throughout the process to ultimately inform the overall findings. During this process of data 
reduction, researchers characterized the prevalence of responses, examined differences among 
groups, and identified key findings and themes related to the research questions. 


Results 


Baseline Equivalence 

We did not find statistically significant differences in the large majority of background 
characteristics across households residing in IN catchment areas and control households. Child 
and household-level characteristics, including district of residence, household perceptions of 
poverty, and household distance from school were similar across treatment and control 
households. Overall, almost half of the sample was age 8 years or older, and slightly more than 
half of the sample was male. Most of the sample resided in Petauke District (compared with 
Katete or Sinda) and considered itself to be “very poor.” More than half of the sample reported 
that the child caregiver had attended school, and the average distance from school was 
approximately 0.77 kilometers. We report descriptive statistics on sample background 
characteristics by treatment and control areas in Table 2. 


Although we found some statistically significant differences between treatment and control 
households, only the difference in household distance from school was larger than 0.25 
standard deviations. A study by Imbens (2015) indicates that statistically significant differences 
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that are smaller than 0.25 standard deviations are unlikely to cause bias. However, we did 
observe some small but statistically significant differences when comparing sample means. For 
example, children in treatment households were less likely to be age 8 years or older (48%) 
compared with control households (53%). We also found that children in control group scored 
higher on baseline EGMA scores than children in the treatment group. All regressions 
estimating the ITT and TOT effects control for variables that showed some imbalance at 
baseline to generate more precise estimates. Specifically, we control for household distance 
from school, baseline EGMA score, and child age at baseline. 


Table 2. Baseline Equivalence 


WETEr] ¢) (13 Mean N1 Mean N2 Diff SE p-Value | Difference 


Background characteristics 


eee ye 0.48 | 888 812 0.02 0.08 -0.08 
female 

Index child was 8 

years old or older 0.53 888 0.48 812 0.03 -0.11 
at baseline 

Caregiver had 

Resided in Katete Soc mm fem pon 


0.13 


> em on | on | 009 


District O08 


-0.11 


Resided in Sinda 

District ? 

Household 

considered itself : 888 812 
to be nonpoor 


0.01 0.61 -0.03 


Household 
considered itself 
to be moderately 
poor 


888 812 0.02 0.03 


812 0.03 0.53 -0.04 


0.05 


Household 
considered itself 888 


to be very poor 
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WETer) 9) (43 


Household 
distance from 
school (km) 


Primary outcomes 


Zambian 
Achievement Test 
(% correct) 


Early Grade 
Reading 
Assessment (% 
correct) 


Early Grade 
Mathematics 
Assessment (% 
correct) 


Oral vocabulary 
(% correct) 


Note. Standard errors (SE) clustered at the school level. 


Attrition and Program Take-Up 

Midline Attrition—During the midline survey, several logistical challenges delayed data 
collection activities. First, we experienced limited access to areas due to poor road conditions 
and heavy rains. Second, poor network connectivity prevented continuous communication with 
teams working in remote areas. Third, it was challenging to track households and children who 
migrated after the baseline. We tracked and administered surveys with 91.61% (N = 1,700) of 
the 1,865 children in the baseline sample. Table 3 below presents a summary of the response 
status by district. The table classifies each household into one of five categories: (1) Complete 
interview, (2) Partially complete interview, (3) Noncontact, (4) Refusal, and (5) Other. We 
provide a detailed definition for each category under Table 3. 


Table 3. Summary of Response Status by District 


Interview Response se Total Number 


Status of Interviews 
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Interview Response ee Total Number 


Status of Interviews 


= 


[Noncontact =| 2. 2.68% | | 2% 6. | 6.89% | 


2 — Soa ea 


Note. “Complete interview” means that both the primary caregiver and the index-child were interviewed by an 
enumerator. “Partially complete interview” means that the primary caregiver was interviewed but not the index- 
child. “Noncontact” means that both the primary caregiver and the index-child were not interviewed, even though 
the household from baseline was located. “Refusal” means that the household refused to participate in the survey. 
Most of the interviews marked as “Other” either relocated or could not be identified using the information 
collected at baseline. 


For unidentified households, we used a combination of three approaches interchangeably to 
pinpoint the location of the household: (1) a snowballing tracking approach that involved 
finding households by asking other members of their community where they were; (2) a GIS 
approach that involved using GPS coordinates from baseline; and (3) an approach that relied on 
using telephone numbers collected at baseline to contact missing households. The data 
collection teams started prioritizing these techniques on March 15, 2019. Combining these 
approaches helped to reduce the attrition rate. We succeeded in tracking 91.61% of the 
households we interviewed during the baseline data collection. 


We found no difference in baseline characteristics between children in the treatment and 
control groups who remained in the study. That is, the reduction in sample size did not cause 
any differential attrition, thereby preserving the study’s internal validity created through 
randomization. We also did not find statistically significant differences in baseline values of 
EGRA, EGMA, ZAT, and Oral Vocabulary scores between children who remained and children 
who withdrew from the sample. Table 4 below shows our test for differential attrition. 


Table 4. Differential Attrition 


Nonattrited rN adaie-te| Difference Test Std. Mean 


VETat} 9) (=13 - Difference 


Treatment 0.48 | 1,700 |} 0.46 | 165 -0.02 0.07 0.81 -0.03 


Zambian Achievement Test 


“ 0.44 | 1,700 | 0.46 165 0.01 0.02 0.53 0.05 
(% correct) 
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Nonattrited Attrited Difference Test Std. Mean 


VWETat}9) (=13 | va | Mean Difference 


pany Glade Beadle 0.05 | 1,700| 0.05 | 165 | 0.00 | 0.00 | 0.75 0.02 
Assessment (% correct) 
Eady Grade Mathematics “| Go7 |) t:700'| 0.07, | 165 || 000) oon || oes 0.02 
Assessment (% correct) 
Oral Vocabulary (% correct) | 0.60 | 1,700 | 0.64 165 0.03 0.03 0.29 0.11 


Note. Standard errors (SE) clustered at the school level. 


Impacts on School Enrollment and Program Take-Up—An analysis of school enrollment found 
sizeable, and statistically significant, impacts on school enrollment. In the treatment areas, 62% 
of the eligible children enrolled in school, while 55% of the children in control areas enrolled in 
school. We estimated statistical tests on the differences across children who enrolled and those 
who did not and found several significant differences between the two groups, which we report 
in Figure 4. 


First, children who scored slightly higher on EGRA, EGMA, ZAT, and the Oral Vocabulary 
assessment at baseline were more likely to enroll in school. Second, girls were more likely than 
boys to enroll in school. Third, the results suggested that children in households with higher 
socio-economic status were more likely to enroll in school. Parents of children who enrolled in 
school had higher levels of education, and their households had higher levels of assets and food 
security compared with parents and households of children who did not enroll in school. We 
did not, however, find large differences in age, geography, birth order, household size, 
ethnicity, and language between enrolled and nonenrolled children. Tables A-1 through A-3 in 
Appendix A of this report provide detailed statistics regarding these samples. 
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Figure 4. Standardized Mean Differences Between Enrollees and Nonenrollees 


HH Food Insecurity Access Scale 
Sinda district =z 

Language: Nsenga = 

Ethnicity: Nsenga i 

Petauke district Oo 

8 years or older | 

Biological child 

First child 

Ethnicity: Chewa 

Household considers itself to be non poor 
Household size 

Language: Chewa 

Katete district 

Oral Vocabulary baseline score 
EGMA baseline score 

EGRA baseline score 

HH Asset Index 

Caregiver attended school 
Highest education for male parent 


ZAT baseline score 


Female 


-0.21 -0.01 0.19 


oO 
ww 
WO 


0.59 


We also found several statistically significant differences between children enrolled in IN 
schools and children who lived in the treatment areas but did not enroll in IN schools, 
suggesting that treatment effects on the treated applied to a sample with different 
characteristics than ITT effects. We estimated a multivariate regression model to predict the 
likelihood of enrolling in IN school for children residing in treatment areas based on 
demographic and household characteristics as well as baseline performances on tests. We 
report these results in Table 5. Overall, the results indicate that children enrolled in IN schools 
were significantly more likely to be female, had higher achievement levels on tests at baseline, 
had parents/caregivers with higher education levels, and resided in households with higher 
asset ownership. A multivariate logit regression model showed that the odds of enrollment in 
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IN schools were almost 209% higher for female children compared with male children (p < 0.01) 
and 71% higher for children from households where the caregiver had attended school (p < 
0.01). Odds of enrollment also were significantly positively correlated with baseline scores on 
EGRA and EGMA (p < 0.05). 


Table 5. Enrollment in IN Schools in IN Catchment Areas 


(1) (2) 


Enrollment in IN Schools Logit Coefficients Odds Ratios 


Index child was female 


Index child was age 6 at baseline 


Index child was age 7 at baseline 


Index child was age 8 at baseline 


Index child was a biological child 


Household size 


Highest level of education achieved by adult male member 


Caregiver had attended school 


Petauke District 


Sinda District 


Zambian Achievement Test (percentage correct) 


Early Grade Reading Assessment (percentage correct) 
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6) 2) 
Enrollment in IN Schools Logit Coefficients Odds Ratios 
1.591" 4.910° 


Early Grade Mathematics Assessment (percentage correct) 
(0.914) (4.490) 


0.137 1.147 
Oral Vocabulary (percentage correct) 

(0.313) (0.359) 

0.057 1.058 
Index child ethnicity: Other 

(0.621) (0.658) 

-0.624 0.536 
Index child ethnicity: Tumbuka 

(1.141) (0.612) 

-0.347 0.707 

(0.299) (0.212) 


-0.001 0.999 


Index child language: Nsenga 


Index child is the first child 
(0.589) (0.589) 


0.130 1.139 
Index child is the second child 

(0.513) (0.584) 

-0.103 0.902 
Index child is the third child 

(0.587) (0.530) 

0.077 1.080 
Index child is the fourth child 

(0.587) (0.634) 

-0.005 0.995 
HH Food Insecurity Access Scale 

(0.016) (0.016) 


0.319""" 1.376 
HH Asset Index 

(0.094) (0.130) 

-0.140 0.869 
Baseline HH distance from school (Km) 

(0.107) (0.093) 


“™p<0.01 


Note. Standard errors clustered at school level and reported in parentheses. * p < 0.10, “p< 0.05, 


AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 30 


Midline Report for the Mixed-Methods Cluster-Randomized Controlled Trial of 
Impact Network’s eSchool 360 Model in Rural Zambia 


Impact Evaluation Results 


Primary Outcomes—ZAT, Reading, Mathematics, and Oral Vocabulary 


ITT Effects on Complete Sample 

Compared with baseline, we found some increases in the difference in test scores of treatment 
and control children 14 months after the program. Figure 5 shows the mean percentage scores 
(as % correct) across treatment and control groups in the two rounds of testing along with 95% 
confidence intervals. 


Results from the formal ANCOVA framework showed statistically significant and positive effects 
of residing in the IN catchment areas on all primary test scores (see Table 6). The findings 
indicate that residing in IN catchment areas (treatment assignment) led to a 0.16 standard 
deviation or 3.1 percentage points increase in ZAT scores (p < 0.05), a 0.40 standard deviation 
or 3.5 percentage points increase in EGRA scores (p < 0.01), a 0.22 standard deviation or 4.9 
percentage points increase in EGMA scores (p < 0.05), and a 0.25 standard deviation or 6.0 
percentage points increase in Oral Vocabulary scores (p < 0.01). ”° 


These results appear to suggest a larger effect on EGRA than on EGMA outcomes, even though 
we found similar impact estimates when examining effects in actual scale (percentage correct). 
We found impact estimates of 3.5 percentage points for EGRA and 4.9 percentage points for 
EGMA outcomes, indicating greater variation in reading compared with mathematics 
achievement among the population. Specifically, the EGRA scores for children in the IN 
catchment areas were slightly more skewed to the right, while the distribution of EGMA scores 
was similar across treatment and control households. 


Table 6. ITT Effects on Primary Outcomes 


VARIABLES % Score 


0.031*** | 0.158*** | 0.035*** | 0.404*** |0.049***) 0.219*** | 0.060*** | 0.251*** 
Treatment 


oe 0.006 | 0.006 | 0.000 | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 


7 In addition to the overall scores, we also found that residing in Impact Network catchment areas increased the likelihood of 
obtaining a positive score (or a non-zero score) on EGRA by 4.7 percentage-points, on EGMA by 1.7 percentage-points, and on 
Oral Vocabulary by 2.2 percentage points. See Table D-1 in Appendix D of this report. 

8 All results also show statistically significant effects after adjusting standard errors based on randomization inference as shown 
in the adjusted p-values. 
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EGRA- % 
Score % Score 


5 oe 0.007 0.007 0.000 0.000 0.001 0.001 0.000 0.000 
randomization 
inference 


1,688 1,688 1,688 1,688 1,688 1,688 1,688 1, | 1688 | 
0.054 0.054 0.056 0.056 0.068 0.068 0.047 0.047 


Control group 
mean 0.548 0.0766 0.210 0.714 


Note. Standard errors clustered at school level and reported in parentheses. * p < 0.1; ** p< 0.05; *** p< 0.01. 
OV = Oral Vocabulary 
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Figure 5. Mean Primary Test Scores (% Correct Responses) Across Treatment and Control Groups 
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Note. N = 1,700. Across all populations, we found that the difference in scores of treatment and control children became larger and more significant at midline 
compared with baseline. 
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Below, we provide several examples from the four tests to show how ITT effects translated into 
differences in learning outcomes. 


1. ZAT: On average, children living in the IN catchment areas scored 3.1 percentage points 
higher than other children on tasks such as: 


a. Recognizing the letters Fl among a list of letter groups that included IP, TT, Tl, and Fl. 


b. Recognizing and pointing to the letter R when shown a group of pictures that included a 
lion, a giraffe, a rhinoceros, and the letter R. 


c. Pointing to a picture of an object that began with the same sound as another object 
among a group of objects. 


2. EGRA: On average, children living in IN catchment areas scored 3.5 percentage points higher 
than other children on tasks such as: 


a. Answering questions based on reading a story about a friendship between a child and a 
snake. 


b. Determining where a sentence began, in which direction they were to read, and where 
to go when they reached the end of a line. 


c. Answering questions based on listening to a story about a child going to school for the 
first time during the rainy season. 


3. EGMA: On average, children living in IN catchment areas scored 4.9 percentage points 
higher than other children on tasks such as: 


a. Recognizing numbers such as 9, 97, or 468. 


b. Identifying which number was missing from a set of patterns (for example, from a 
pattern of 10, 20, ..., and 40, the child was asked to identify the missing number). 


c. Adding two numbers, such as 67 + 25. 


4. Oral Vocabulary: Children living in IN catchment areas scored 6.0 percentage points higher 
on tasks like listening to a word given to them and pointing to a picture in a booklet that 
best represented that word. Figure 6 is an example given to children participating in this 
study: An enumerator may, for instance, have said the word “lion” and then asked a child to 
point to the image of a lion. 
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Figure 6. Example of a Question From the Oral Vocabulary Test 


Heterogeneous Effects 

We did not find substantial variation in the effects of residing in IN catchment areas by 
subgroups. Figure 7 shows ITT effects from different regressions including interactions between 
treatment assignment and: (a) child gender (female versus male), (b) child age at baseline (age 
6 years or 7 years versus age 8 years or 9 years), (c) caregiver’s education (whether the 
caregiver ever attended school), (d) household district (Katete, Sinda, or Petauke), (e) child 
score on baseline ZAT (above versus below median score), (f) child score on baseline EGRA 
(above versus below median score), and (g) child score on baseline EGMA (above versus below 
median score). The figure shows that almost all interaction terms were statistically insignificant, 
suggesting that the eSchool 360 program produced similar benefits for each of the subgroups in 
our sample. The results did suggest that residing in an IN catchment area had a larger impact on 
EGRA scores at midline for children who scored above median score on EGMA and Oral 
Vocabulary at baseline, but these results were only marginally significant (p<0.10) after 
correcting for multiple comparisons, as shown in Tables B-1 to B-8 in Appendix B of this report. 
In general, we need to exercise some caution in interpreting these results because we may not 
have adequate statistical power to detect small heterogeneities in the effect sizes with 
sufficient precision. 


TOT 

To estimate the TOT effects, we first estimated the impact of attending school at least three 
times in the week before the survey as an indicator of consistent attendance at school. Figure 8 
shows the difference in test scores from baseline to midline for these children, referred to as IN 
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attendees, compared with non-attendees. We find substantially larger increases in test scores 
for attendees across all four tests compared with non-attendees. 


Figure 7. Heterogeneous ITT Effects 
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Figure 8. Mean Primary Test Scores (% Correct Responses) Across IN Attendees and Non-Attendees 
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Note. N = 1,700. The term “Attendees” refers to children who were reported to have attended an IN school at least three times in the last week. The term 
“Non-attendees” refers to all others. 
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We also present results from estimating the impact of having ever enrolled in an IN school over 
the year prior to the survey. We show all TOT effects formalized in an instrumental variable 
framework in Table 7, where column (1) shows the relationship between IN school assignment 
and IN school enrollment; and columns (2) to (9) show second-stage estimates reflecting the 
treatment effects on the treated. 


Table 7. TOT Effects on Primary Outcomes 


EGRA- % 
Score 


SF a ee eed ele eee eee 
7 A 


0.063*** | 0.325*** | 0.072*** | 0.834*** 10.101***! 0.450*** | 0.124*** |0.516*** 


[rem [eaom | wo | ose [com | 0325 | (oar) | 9) 
ee 


1,688 1,688 1,688 1,688 1,688 1,688 1,688 1,688 1,688 
0.331 0.107 0.107 0.140 0.140 0.147 0.147 0.082 0.082 


Co A 
A 
P| enn | ene [eon | psn [em [oxo | pmo [een 


R- R-squared | | 0.435 435 | 0.095 095 | 0.095 | 095 | 0.122 122 | orga | 122 0. | 0.130 | 0.130 | 130 | 0.079 | 079 0. | 0.079 | 


Control group 
mean 0.526 0.0689 0.181 0.707 


Note. Standard errors clustered at school level and reported in parentheses. * p < 0.1; ** p< 0.05; *** p< 0.01. 


TOT effects showed that attending an IN school at least three times in the week before the 
survey had a significant positive impact on all primary test scores. Specifically, we found that 
that proxy for active participation in the program led to an increase of 0.32 standard deviation 
or 6.3 percentage points in the ZAT score, a 0.83 standard deviation or 7.2 percentage point 
increase in the primary EGRA score, a 0.45 standard deviation or 10.1 percentage point 
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increase in the primary EGMA score, and a 0.52 standard deviation or 12.4 percentage point 
increase in the Oral Vocabulary score. We found that children residing in an IN school 
catchment area were 49 percentage points more likely than other children to have attended an 
IN school at least three times in the week. 


The TOT effects remain substantial but are smaller when we use ever enrolling in an IN school 
as a proxy for participation in the program. We found that enrollment in IN schools led to an 
increase of 0.26 standard deviations or 5.2 percentage points in the ZAT score, a 0.68 standard 
deviation or 5.8 percentage points increase in the primary EGRA score, a 0.37 standard 
deviation or 8.2 percentage points increase in the primary EGMA score, and a 0.42 standard 
deviation or 10.1 percentage points increase in the Oral Vocabulary score. The effects are likely 
smaller than in the previous analysis because children who were ever enrolled in IN schools 
may not have consistently attended school, limiting the benefits of their enrollment. Indeed, 
the development economics literature suggests that enrolling in school is not always a good 
proxy for attending school (Alderman, Gilligan, and Lehrer, 2012). 


Overall, the TOT estimates are larger than the ITT estimates because the ITT effects compared 
all children in IN catchment areas with other children irrespective of whether they attended an 
IN School. The ITT estimates are, therefore, a weighted average of effects on children who were 
and who were not enrolled in school, while the TOT results estimated the impact of attending 
an IN school (proxied by either attending school at least three times in the week before the 
survey or being ever enrolled in IN schools). 


Equivalent Years of Schooling 


We found effect sizes for EGRA and EGMA assessments that were equivalent to 1.88 additional 
years of education for reading and 1.03 additional years of education for mathematics if we 
transformed the effect sizes of 0.40 standard deviations on EGRA scores and 0.22 standard 
deviations on EGMA scores and extrapolated results of learning gains from Grade 1 to Grade 12 
from a sample of representative low- and middle-income countries (Evans & Yuan, 2019). The 
equivalent years of schooling were based on the finding that 4.7 years of schooling was 
equivalent to a one standard deviation improvement in reading proficiency in a pooled sample 
of five representative low-and middle-income countries. When compared directly against the 
one-year learning gains in the control group in the original test scale and units, the ITT 
estimates are equivalent to 1.2 additional years of education for reading and 0.37 additional 
years (or 4.5 additional months) of education for mathematics. 
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Reading and Mathematics Subscores 


Subscores on Early Reading Assessment (EGRA) 

Previous experience in Zambia suggests that a majority of first-grade students score low on 
their EGRA and EGMA tests, with many zero scores, indicating that we may find larger effects 
on the easier EGRA and EGMA subcomponents that are administered in the early stages of the 
assessment than on the more complex EGRA subcomponents that are administered in the later 
stages of the assessment. However, we found that living in IN catchment areas had a significant 
positive impact on all but one EGRA subcomponent, which suggests that the eSchool 360 model 
leads to improvements in easier and more complex EGRA subtasks. Figure 9 shows descriptive 
differences in mean scores from baseline for the treatment and control groups. 


Columns (1) through (3) in Tables 8 and 9 show statistically significant ITT effects on seven out 
of eight subtasks of the EGRA assessment: orientation to print, letter sound knowledge, 
nonword decoding, oral passage reading (basic and higher levels), and reading comprehension 
(basic and higher levels). The largest effect in percentage points was for orientation to print (8.7 
percentage points), while most of the other effect sizes ranged between 2 and 4 percentage 
points. We did not find a statistically significant effect on listening comprehension. The results 
were robust to correcting for multiple comparisons using the Benjamini-Hoschberg method (as 
shown in the adjusted q-value in column (3)). 


As expected, we find much larger differences when comparing children who attended IN 
schools at least three times in the last week compared with other children (Figure 10). 
Estimates of TOT effects (based on the proxy of attending school at least three times in the last 
week) show a strong positive impact of attending IN schools on all EGRA subtasks except 
listening comprehension (Tables 8 and 9, columns (4) though (6)). We found particularly large 
effect sizes for some subtasks—specifically, having enrolled in an IN school increases basic oral 
passage reading performance (level 1) by 1.1 standard deviations or 7.3 percentage points, and 
higher-level oral reading performance by 1 standard deviation or almost 5.8 percentage points 
(level 2). 
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Figure 9. Mean EGRA Subscores (% Correct Responses) Across Treatment and Control Groups 
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Note. N = 1,700. 
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Figure 10. Mean EGRA Subscores (% Correct Responses) Across IN Attendees and Non-Attendees 
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Note. N = 1,700. Attendees are children who were reported to have attended an IN school at least three times in the last week. Non-attendees include all 


others. 
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Table 8. ITT and TOT Results on EGRA Subscores (Standard Deviation) 


pa | a] ow | w | @ | 


ANCOVA Estimates: ITT Effects IV 2SLS Estimates: TOT Effects 


Coef. | Unadjusted | Adjusted Coef. | Unadjusted | Adjusted 
Subsample (SE) p-Value q-Value (13) 


0.202*** 0.003 0.003 
Orientation to Print 


(0.064) 


0.305*** 0.001 0.001 


(0.086) 


0.428*** 0.000 0.000 
0.524*** 0.000 0.000 
Oral Passage Reading level 1 

0.404*** 0.000 0.000 
Reading Comprehension level 1 

0.485*** 0.000 0.000 
Oral Passage Reading level 2 

OB 0.000 0.001 
Reading Comprehension level 2 

0.068 0.229 0.229 0.140 0.217 0.217 

Listening Comprehension 


Note. Standard errors clustered at school level and reported in parentheses. TOT effects are based on having 
attended an IN school at least three times in the last week. * q < 0.1; ** q < 0.05; *** q < 0.01 


0.417*** 


(0.186) 


Letter Sound Knowledge 


Nonword Decoding 
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Table 9. ITT and TOT Results on EGRA Subscores (Percent Correct) 


in Poe Boe eae 
ANCOVA Estimates: ITT Effects IV 2SLS Estimates: TOT Effects 


Unadjusted | Adjusted Unadjusted | Adjusted 
p-Value q-Value 


0.087*** 0.003 0.003 |0.181*** 0.001 0.001 
Orientation to Print 
(0.028) | oss) | | 
0.033*** 0.001 0.001 |0.067*** 0.000 0.000 
Letter Sound Knowledge 
(0.009) | os) | | 
0.030*** 0.000 0.000 |0.062*** 0.000 0.000 
Nonword Decoding 
ooo | (0.014 
0.035*** 0.000 0.000 |0.073*** 
Oral Passage Reading Level 1 
(0.008) | (0.015) 
0.041*** 0.000 0.000 |0.085*** 
Reading Comprehension Level 1 
(0.010) | (0.020) 
0.028*** 0.000 0.000 |0.058*** 
Oral Passage Reading Level 2 
(ooo | (0.018 
0.031*** 0.000 0.001 |0.063*** 
Reading Comprehension Level 2 
(0008) | | (0.017) 
0.026 0.229 0.229 0.054 
Listening Comprehension 


Note. Standard errors clustered at school level and reported in parentheses. TOT effects are based on having 
attended an IN school at least three times in the last week. * q < 0.1; ** q < 0.05; *** q < 0.01 


Subscores on Early Grade Mathematics Assessment (EGMA) 

Figure 11 shows descriptive differences in mean scores from baseline for the treatment and 
control groups. We see particularly large increases in scores on rational counting, number 
recognition, and pattern completion. 
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Figure 11. Mean EGMA Subscores (% Correct Responses) Across Treatment and Control Groups 
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The ITT estimates showed statistically significant effects on most of the EGMA subtasks, with 
larger effects on easier EGMA subtasks. The effect sizes in standard deviations for EGMA were 
smaller than on the EGRA subtasks, even though effects in actual scale were slightly higher on 
EGMA subtasks, again reflecting the more positively skewed distribution of EGRA scores for 
children in treatment areas. Columns (1) through (3) of Table 10 show that being assigned to IN 
had a small but significant positive impact on rational counting, number recognition, quantity 
discrimination, and pattern completion. We found no statistically significant impacts on oral 
counting, word problems, and two levels each of addition and subtraction questions. 


Again, we see much higher scores on all EGMA subtasks for the sample that reported having 
attended an IN school at least three times in the previous week (Figure 12). However, estimates 
of TOT based on the proxy of attending school at least three times in the week before the 
survey did not show statistically significant effects for most EGMA subtasks except rational 
counting (a 0.43 standard deviation increase), number recognition (a 0.84 standard deviation 
increase), and quantity discrimination (a 0.42 standard deviation increase). Columns 4-6 of 
Table 10 depict these results. 
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Table 10. ITT and TOT Results on EGMA Subtask Scores (Standard Deviation) 


sie ae ee ee 
ANCOVA Estimates: ITT Effects IV 2SLS Estimates: TOT Effects 


Coef. | Unadjusted | Adjusted Unadjusted | Adjusted 
Subsample (SE) p-Value (PAVE (U =] 


0.075 0.136 0.170 0.154 0.124 0.155 
Oral Counting 

(oso) | (200) | | 

0.210*** 0.000 0.001 | 0.433*** 0.000 0.000 
Rational Counting 

(ose) | ooo) | | 
0.408*** 0.000 0.000 _ | 0.840*** 0.000 0.000 
Number Recognition 

(ors) | on) | | 

0.205*** 0.001 0.003. | 0.423*** 0.000 0.001 
Quantity Discrimination 

(0.060) | oa) fo | 

0.010 | 0.456*** 0.001 0.003 

(0.143) 


Pattern Completion 
(0.074) 


O22 es 0.004 
0.090 0.112 0.160 0.185 0.095 0.136 
Word Problems 


(o0se) | oa | | 

0.110 0.062 0.124 0.226 0.046 0.092 
Addition Questions (Level 1) 

(ooss) | oa) | | 

0.119 0.085 0.141 0.245 0.068 0.113 
Addition Questions (Level 2) 

(ooss) | asa) | | 
Subtraction Questions (Level 0.077 0.168 0.187 0.159 0.146 0.162 
2 (ooss) | oso) | | 
Subtraction Questions (Level 0.083 0.244 0.244 0.171 0.227 0.227 


Note. Standard errors (SE) clustered at school level and reported in parentheses. TOT effects are based on having 
attended an IN school at least three times in the last week. * q < 0.1; ** q < 0.05; *** q < 0.01 
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Table 11. ITT and TOT Results on EGMA Subtask Scores (Percent Correct) 


SA See Ra aa ri 
ANCOVA Estimates: ITT Effects IV 2SLS Estimates: TOT Effects 


Coef. | Unadjusted | Adjusted Unadjusted | Adjusted 
Subsample (13) p-Value q-Value 


0.019 0.136 0.170 0.038 0.124 0.155 
Oral Counting 
oo | | oor) | | 
0.041*** 0.000 0.001 | 0.084*** 0.000 0.000 
Rational Counting 
(oo) | oo) | | 
0.087*** 0.000 0.000 | 0.178*** 0.000 0.000 
Number Recognition 
oor) | | (0030) | | 
0.065*** 0.001 0.003 | 0.135*** 0.000 0.001 
Quantity Discrimination 
oi) | | (os) | | 
0.047*** 0.004 0.010 | 0.098*** 0.001 0.003 
Pattern Completion 
(0.016) (0.031) 


oo) | (oo) | | 

0.046 0.062 0.124 0.094 0.046 0.092 
Addition Questions (Level 1) 

(ooze) | oo) | | 

0.021 0.085 0.141 0.044 0.068 0.113 
Addition Questions (Level 2) 

oo) | | (oor) | | 
Subtraction Questions (Level 0.030 0.168 0.187 0.061 0.146 0.162 
Subtraction Questions (Level 0.015 0.244 0.244 0.031 0.227 0.227 
2 (ois) | | (oor) | | 


Note. Standard errors (SE) clustered at school level and reported in parentheses. TOT effects are based on having 
attended an IN school at least three times in the last week. * q < 0.1; ** q < 0.05; *** q < 0.01 


0.030 0.112 0.160 0.062 0.095 0.136 
Word Problems 
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Figure 12. Mean EGMA Subscores (% Correct Responses) Across IN Attendees and Non-Attendees 
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Note. N = 1,700. Attendees are children who were reported to have attended an IN school at least three times in the last week. Non-attendees include all 
others. 
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In line with our quantitative finding that the IN program had statistically significant positive 
impacts on literacy, qualitative data showed perceived improvements in children’s literacy 
according to parents, teachers, and IN staff. Parents and program staff noted that children at IN 
schools read better than their counterparts attending government-run schools and started to 
read earlier (in Grade 1). One parent from Kabanga School commented to this end: “...when you 
take the children in a government school...they have difficulties in knowing how to read; but 
with this [IN] school they easily know how to read.” Children, for their part, shared how they 
learned to read through activities involving the projector. When asked why they liked the 
projector, one student from Bvunda School said, “We /earn how to read through it. We get 
words.” Classroom observations confirmed that children did various literacy-related activities 
using technology, including listening to stories via the projector and reading stories in their 
tablets. Interestingly, however, respondents did not report perceived changes in numeracy 
resulting from the IN program. 


Mechanisms of Impact 

This section examines impacts on intermediate outcomes to assess potential mechanisms that 
drove the impacts on EGRA, EGMA, ZAT, and Oral Vocabulary assessments. For this analysis, we 
examined impacts on intermediate outcomes followed by an analysis of the correlation 
between intermediate outcomes and EGRA, EGMA, ZAT, and Oral Vocabulary scores. 


We first assessed the program impacts on school enrollment and attendance to determine 
whether increased schooling may have contributed to the positive impacts. Second, we 
assessed the effects on parents’ perceptions of school and education quality by estimating 
impacts on 4-point Likert-scale questions related to parents’ general perceptions on the quality 
of education as well as to more specific attitudes related to IN’s activity-based curriculum, the use 
of technology in the classroom, and teachers’ pedagogical practices. We examined program 
impacts on children’s level of engagement in the classroom, the time devoted to activity-based 
learning activities, the use of technology in the classroom, and children’s interaction with 
teachers. Third, we estimated impacts on child development, which we measured by using an 
index constructed based on answers of parents about the behavior of their children. Figure 13 
plots correlations between each of these factors and test scores across the four assessments; 
and Tables C-1 and C-2 of Appendix C show results from multivariate regressions depicting the 
correlations between multiple factors and test scores. 
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Figure 13. Intermediate Outcomes and Primary Test Scores 
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School Enrollment and Attendance 

We found that living in IN catchment areas increased the likelihood of school enrolment by 
almost 8 percentage-points. Figure 13 shows that school enrollment is positively correlated 
with test scores, suggesting that increased likelihood of enrollment was likely an important 
mechanism for improvements in learning outcomes due to the eSchool 360 model. We also 
found that residing in IN area led to an increase of 0.36 days of weekly attendance (or 1.6 days 
in a month), on average (Table 12). Figure 13 shows that learning outcomes on all four tests 
jump beyond one day of weekly attendance but increase at a modest pace beyond that. This 
may be indicative of a significant—although small—mediating effect of school attendance. 
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Table 12. ITT Effects on School Enrollment and Attendance 


a 


Number of 
Number of Days Age at 
Taleo) |(=Xe| Enrolled DEW Attended Age at Enrollment 
VARIABLES (yes/no) (SMD) Attended (SMD) Enrollment (SMD) 


0.079** 0.159** 0.358** 0.162** -0.096** -0.094** 
Treatment 


(0.038) (0.075) (0.158) (0.071) (0.038) (0.037) 


0.021 0.021 0.024 0.024 0.800 0.800 


Note. Standard errors clustered at school level and reported in parentheses. * p < 0.1; ** p< 0.05; *** p< 0.01. 


Results from interviews and focus groups also showed that parents, teachers, and program staff 
noted high student attendance and enrollment in IN classes. A member of IN’s staff noted that 
the “average student attendance was about 90%.” A few respondents mentioned that children 
were motivated to attend class due to interest in using tablets and technology in the classroom. 
A teacher explained, “/ think we have the iSchool, where they [students] come to school and 
learn using the projector and tablets. | think they feel it’s a plus because they feel good to come 
to school and learn the new method of teaching [so] they attend classes.” Program staff felt that 
attendance and enrollment was higher in IN classes compared with grades that did not offer 
the eSchool 360 program. 


Parents and program staff mentioned that some parents were less motivated to enroll their 
children in classes if they did not receive the eSchool 360 intervention. According to 
respondents, IN transitioned from implementing the program in Grade 1 to Grade 2 during the 
2018-19 school year. As a result, an IN staff explained that “some parents were hesitant to 
enroll children in a nonprogram class.” This was especially relevant to Grade 1, which previously 
had benefited from the eSchool 360 program. Several parents expressed concern relative to 
school quality and cost in the absence of the of eSchool 360 intervention. 


While IN staff agreed that overall student attendance was “pretty high,” they acknowledged 
seasonal trends in attendance and indicated attendance was lower during rainy season. IN staff 
reported that in the rainy season, some children were still working on their family farms, 
herding cattle (boys only) or caring for younger siblings while parents worked on the farm, or 
were unable to cross rising rivers due to rain. 
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Interestingly, we found that age at enrollment decreased significantly because of the 
introduction of the IN Schools. Table 12 shows that students in the IN catchment areas were, on 
average, 0.09 years younger than students in control schools. This difference is statistically 
significant and indicates that parents may have decided to send children to school at an earlier 
age because of improved perceptions of education quality or because of the zero costs of 
attending IN schools. However, the decreased age at enrolment likely did not contribute to 
improvements in learning outcomes, considering that age at enrollment did not show a 
monotonically positive correlation with learning outcomes (Figure 13). 


Child Development 


Although we found significant correlations between child development (as measured through a 
child cognitive skills scale) and learning outcomes (Figure 13), the ITT estimates did not show 
statistically significant effects of the eSchool 360 model on child development outcomes (Table 
13). All point estimates were positive, but the impact estimates were not statistically 
significant. This finding indicates that the positive impacts on EGRA, EGMA, ZAT, and Oral 
Vocabulary assessments were likely not driven by improvements in child development. 


Table 13. ITT Effects on Other Intermediate Outcomes 


a 


Caregiver Caregiver 
Child Caregiver | Perception/ | Perception/ 
Child Developmen Caregiver | Satisfaction | Engagemen | Engagemen 
Developmen t Scale Satisfaction RY or-)( t 
VARIABLES La Yor (-] (SMD) RYor- (1 (SMD) RYor=} (= 6:11] B)) 


0.242 0.084 0.299** 0.284** 0.845*** 0.316*** 
Treatment 


(0.174) (0.061) (0.121) (0.114) (0.239) (0.089) 


R- Rsquared —_| 0. 0.037 | | 0.037 037 ae 047 0. | 0.047 | ee 033 | 0.033 | 033 


Control group 
mean 17.94 1.189 7.729 


Note. Standard errors clustered at school level and reported in parentheses. * p < 0.1; ** p< 0.05; *** p< 0.01. 


Parents’ Satisfaction With School and Education Quality 

We found statistically significant effects on parents’ perceptions of school and education 
quality. Table 13 shows that caregivers in the IN catchment areas reported a 0.30-point higher 
satisfaction with education than caregivers in the control group (p < 0.05). We based these 
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impact estimates on interviews with caregivers of children who enrolled in any school in the 
past or in the current year. 


We also found positive correlations between parents’ perceptions of school education quality 
and learning outcomes, suggesting that improvements in the quality of education may have 
contributed to the positive effects of the eSchool 360 model on EGRA, EGMA, ZAT, and Oral 
Vocabulary assessments (Figure 13). 


Parents, teacher supervisors, students, and program staff shared the belief that IN teachers 
provided students with high-quality education. Respondents associated the provision of quality 
education with knowledgeable teachers who used innovative teaching methods, constant 
teacher attendance, teacher follow-up when students were absent, and improvements in 
student performance. 


Qualitative results showed that parents believed that IN teachers were well-trained and able to 
apply effective teaching methods including the use of technology in the classroom. Several 
parents mentioned that IN teachers benefited from trainings that improved their teachings 
skills, with one parent noting, “Impact teachers go for refresher courses frequently. So from that 
Impact has helped our teachers to become very educated teachers.” Parents also appreciated 
the use of technology in the classroom and felt that it enhanced the learning process. One 
parent explained how teachers used tablets to improve 
lesson delivery: “What I saw in the way they use it [the 
tablet]—it has lessons in math. When they start 
teaching, the children follow. There are different types of 
lessons. And that’s why the program should not end here 
but should continue, because the children are learning 
well.” Several teacher supervisors also noted that 
trainings increased teachers’ capabilities and confidence 
in the classroom. 


Parents, students, teacher supervisors, and program 
staff agreed that IN teachers were always present in 


class. Respondents noted that if a teacher was absent, 


iSchool tablets. 


another teacher would cover the class. One parent 

explained this practice: “She [the teacher] comes daily, 

but when she is not feeling well she calls another teacher here to come and teach the children.” 
Teacher supervisors and program staff agreed with this perception and noted that strong 
monitoring practices encouraged teachers to attend class regularly. Program staff explained 
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that teacher supervisors and operations managers regularly monitored teacher attendance. In 
addition, teachers were required to bring medical notes explaining any absences. Program staff 
also mentioned that constant teacher attendance differentiated IN from government schools in 
which teachers were frequently absent. 


Parents, teacher supervisors, and program staff believed that improvements in student 
performance demonstrated that teachers provided high-quality education to students. 
Respondents frequently noted that students enrolled in eSchool 360 classes had strong literacy 
skills compared with students enrolled in similar or higher grades in government schools. One 
parent explained this difference: “The children at this school learned how to read from Grade 1; 
but you find a child who is in Grade 4 there [in a government school] but does not know how to 
read.” Teacher supervisors and program staff reported similar stories of students learning to 
read from an earlier age when enrolled in IN classes. 


Parents’ Perceptions of School Engagement 

We found statistically significant effects of the eSchool 360 model on a parental engagement 
scale, but it was unclear whether these impacts contributed to increases in learning outcomes 
because parental engagement and parental satisfaction are highly correlated with each other. 
Table 14 shows that parents reported statistically significantly higher engagement levels in IN 
catchment areas than in control areas. While parental engagement was found to have a 
significant positive correlation with learning outcomes at a bivariate level (Figure 13), this 
relationship was not robust to controlling for parental satisfaction and other factors, as shown 
in Table C-2 in Appendix C, possibly because of the high correlation with parental satisfaction. 


Qualitative results suggested that parents valued the fact that teachers consulted with them 
when a student was absent. One parent recounted frequent communication with her child’s 
teacher whenever the child was absent from school: “They tell us, they call us and they ask us 
why the child isn’t coming to school.” Teachers also mentioned the protocol that they followed 
when students were absent, with one teacher explaining, “If | see that the child has remained 
absent, | go to the parents to visit to see what is happening. Even if the child has been absent 
because of sickness, I'll go visit them to see how the child is feeling.” Parents mentioned that 
active communication with teachers demonstrated IN teachers’ strong commitment 

to teaching. 


Secondary Outcomes 

This section presents program impacts on secondary outcomes, including parents’ aspirations, 
food security, and education expenditures. First, we estimated effects on parents’ aspirations 
with respect to their children’s education and marriage. To measure parental aspirations, we 
asked parents for the bride prices they expected to receive for their daughters, the level of 
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education they would like their children to attain, and their children’s preferred ages at 
marriage. Second, we assessed program impacts on food security and education expenditures. 


Aspirations About Children’s Education and Marriage 

We found no evidence for effects of the eSchool 360 model on parental-level aspirations. Table 
14 shows that we did not find statistically significant effects on the probability parents assigned 
to their children studying beyond the senior secondary level (beyond 12" grade), their 
children’s preferred age at marriage, or expected bride prices for their daughters. 


Table 14. ITT Effects on Caregivers’ Aspirations and Household Food Security 


Child Aspired 
to study 
Ko l=F-] WV £-m vo) min ol =\ Vola le i Ada Wim 4 ol Leite Mae (=n (olUCY-LaLe) (eM celle MMi a elUCY-1are) (eM Yoa Tele) 
Marriage Grade Price Security Related Expenditure 


Zambian Zambian 
VARIABLES SMD |Yes/No| SMD | Kwacha ANVE-Tol a Fe) SMD 


-0.081 | -0.019 | -0.023 |-0.046| -490.963 | -0.105 |1.129**| 0.201** | -13.878** |-0.161** 
Treatment 

(0.249) | (0.059) | (0.038) |(0.077)| (346.806) | (0.074) | (0.479) | (0.085) (6.160) (0.072) 
Observations| 1,625 | 1,625 | 1,619 | 1,619 | 696 | 696 | 1,109 1,109 1,109 
R-squared 0.042 | 0.042 | 0.026 |}0.026}; 0.033 0.033 | 0.048 0.048 0.026 


group mean | 25.98 L! 0.554 zz 2849 _ 11.91 4 55.72 


Note. Standard errors (SE) clustered at school level and reported in parentheses. * p < 0.1; ** p < 0.05; *** p< 
0.01. 


Household Food Security and School-Related Expenditure 

Interestingly, the findings demonstrated positive effects of the eSchool 360 model on the food 
security of households residing in IN catchment areas. We found that treatment households 
scored 1.13 points or 0.20 standard deviations higher on a food security scale than control 
households (Table 14). There are two potential reasons that we will investigate further during 
endline data collection: (1) The effects on food security may have been caused by increased 
food expenditures enabled by reductions in education expenditures due to the free education 
offered by IN (results shown in columns 9 and 10 of Table 14); and (2) The impact may have 
been channeled through an increase in school enrollment if any nutrition initiatives were 
implemented in schools in our catchment areas. These factors will be explored in more detail in 
the next wave of data collection. However, we already found some evidence for statistically 
significant reductions in education expenditures caused by the eSchool 360 model. 
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Process Evaluation Results 


Fidelity of Implementation 

Kils, FGDs, and direct observations revealed that the IN program was implemented with a high 
level of fidelity. Students, parents, IN staff, and teachers themselves reported that IN teachers 
were regularly present at school, used tablets and projectors consistently, employed 
participatory learning approaches such as group works, and followed up with students when 
they were absent. Regarding teacher attendance and follow-up with absent students, one 
Kabanga School parent stated, “The teacher is very good. Any time the children come to school 
there is no time they come back [and say] that the teacher is not there. When the child is absent 
from school the teacher makes a follow-up to us parents to get the reason the child is absent.” 
Qualitative respondents also confirmed that IN schools consistently had the prescribed number 
of tablets—seven pupil tablets and two teacher tablets. 


Classroom observations showed IN teachers using active, participatory pedagogical approaches 
as they were trained to do, such as putting students into small groups and inviting them to 
actively participate in the lesson. For example, teachers called on individual students to come 
to the board to identify different types of punctuation during one observation. Respondents 
also confirmed that IN teachers closely adhered to the prescribed curriculum. Teachers 
followed a curriculum map that showed which lessons to teach and when, and they viewed 
lessons in the tablet and modified them to match the level of their learners. IN staff reported 
that they were fairly strict with newer teachers and asked them to adhere closely to what was 
in the tablet; gradually, however, teachers were given more flexibility to adapt. 


One notable modification to the program was made at the beginning of 2019, when IN staff 
added 30 minutes of literacy-focused activities and 30 minutes of numeracy-focused activities 
to the daily lesson plan. These two 30-minute sessions replaced creative and technology studies 
lessons and were intended to prioritize learning to read, write, and count. Given that children 
were in school for only four hours per day, IN staff reported wanting to place greater emphasis 
on literacy and numeracy to accelerate learning in these two areas. 


Variation by Geography, Culture, and Season 

Cultural and geographic variation across the three IN districts was somewhat limited given that 
all program districts are in Zambia’s Eastern Province, but Petauke is larger and more expansive 
than the other districts. It is not surprising, then, that children in Petauke reported traveling the 
greatest amount of time to reach school (25 minutes each way, on average) compared with 
Katete (17 minutes) and Sinda (11 minutes). 
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During interviews, IN staff shared their perception that girls attended school more frequently 
than boys, and our quantitative findings confirmed this. Of the IN students who attended all 
five of the previous school days, 57.4% were girls and 42.6% were boys. Of the IN students who 
did not attend school any of the five previous school days, 57.5% were boys and 42.5% were 
girls. IN staff also reported during interviews that students who walked more than one hour to 
reach school attended less regularly, and we saw quantitative evidence that the length of time 
required to reach school may have affected attendance. Of the children enrolled in IN schools 
who reported they did not attend school at all during the previous week, 83.1% lived more than 
30 minutes away from their school. Conversely, of the IN children who reported attending 
school all five of the previous school days, 71.1% lived less than 30 minutes from the school. 
These data suggest that longer distances to school may have adversely affected attendance at 
IN schools. 
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Conclusion 


This report presents the midline results of the mixed-methods cluster-RCT on the impact of IN’s 
eSchool 360 model on ZAT, EGRA, EGMA, and Oral Vocabulary assessment scores 14 months 
after starting the program. To determine the impact of the program, we used a cluster-RCT in 
which 63 eligible schools were randomly assigned either to receive IN’s eSchool 360 program 
(30 treatment schools) or not to receive the program (33 control schools). In addition, we 
conducted a process evaluation to determine the fidelity of program implementation. 


The results of our study showed statistically significant ITT effects on EGRA, EGMA, ZAT, and 
Oral Vocabulary assessment scores. Specifically, we found statistically significant effects of 0.16 
standard deviations or 3.1 percentage points on ZAT scores, 0.40 standard deviations or 3.5 
percentage points on EGRA scores, 0.22 standard deviations or 4.9 percentage points on EGMA 
scores, and 0.25 standard deviations or 6.0 percentage points on Oral Vocabulary scores. The 
learning gains were equivalent to 1.88 additional years of education for reading and 1.03 
additional years of education for mathematics if we extrapolate results from a sample of 
representative low- and middle-income countries (Evans & Yuan, 2019). The learning gains 
were equivalent to 1.2 additional years of education for reading and 0.37 additional years (or 
4.5 additional months) of education for mathematics if we compare the ITT estimates in original 
units to the learning gains in the control group. 


TOT effects showed larger impacts on ZAT, EGRA, EGMA, and Oral Vocabulary assessment 
scores for students who started attending the IN schools. Attending an IN school led to an 
increase of 0.32 standard deviation or 6.3 percentage points in the ZAT score, a 0.83 standard 
deviation or 7.2 percentage point increase in the primary EGRA score, a 0.45 standard deviation 
or 10.1 percentage point increase in the primary EGMA score, and a 0.52 standard deviation or 
12.4 percentage point increase in the Oral Vocabulary score when we used attending IN schools 
3 times in the week before the survey as a proxy for regular attendance in IN schools. Enrolling 
in an IN school led to improvements of 0.26 standard deviation or 5.2 percentage points in the 
ZAT score, a 0.68 standard deviation or 5.8 percentage points increase in the primary EGRA 
score, a 0.37 standard deviation or 8.2 percentage points increase in the primary EGMA score, 
and a 0.42 standard deviation or 10.1 percentage points increase in the Oral Vocabulary score 
when we used ever enrolling in IN schools as a proxy for enrollment. 


Evidence suggested that increase in school enrollment and attendance, and improvements in 
the quality of education were likely the main drivers of the positive effects of the eSchool 360 
model on learning outcomes. Children living in IN catchment areas were almost 8 percentage 
points more likely to be enrolled in schools compared to children in the control group, which 
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translated into large increases in school attendance. In addition, among enrollees, caregivers in 
the IN catchment areas reported 0.30 points higher satisfaction with education (ona 
satisfaction scale), and 0.84 points higher level of engagement (on an engagement scale) in 
their child’s education and school activities, compared with the control group. Parents’ 
perceived quality of education and school attendance were also highly and statistically 
significantly correlated with EGRA, EGMA, ZAT, and Oral Vocabulary scores. However, we did 
not find statistically significant effects on child development outcomes. 


In line with the quantitative research, all categories of qualitative respondents (students, 
teachers, parents, and program staff) showed a high perceived quality of education at IN 
schools. Many respondents alluded to their belief that IN schools provided a higher quality 
education than government schools and that teachers were present far more regularly at IN 
schools than at government schools. In agreement with our quantitative findings, qualitative 
respondents also perceived notable improvements in literacy as a result of the IN program, 
although they did not report similar improvements in numeracy. 


Perhaps surprisingly, we also found positive effects of the eSchool 360 model on food security. 
We should exercise caution in interpreting this result, however, because we did not anticipate 
this mechanism. We will examine whether reductions in education expenditures (because of 
the free education offered by IN) enabled increases in food expenditures during the endline 
survey. The midline results already showed statistically significant reductions in education 
expenditures as a result of the eSchool 360 program, suggesting that the program may have 
enabled increases in food expenditures. We did not find statistically significant effects on 
parental-level aspirations, however. The eSchool 360 model did not show statistically significant 
effects on the probability parents assigned to their children relative to graduating from Grade 
12, their children’s age-at-marriage preferences, or the excepted bride price of their daughters. 


A high level of fidelity of implementation likely contributed to the eSchool 360 model’s positive 
effects on learning outcomes. Teachers used the participatory pedagogical approaches on 
which they were trained, adhered to the curriculum prescribed in the curriculum map, and used 
technology (tablets and projectors) as intended. While program implementation did not appear 
to vary based on geography or seasonality, we found evidence that student attendance may 
have been lower during rainy season and lower throughout the year for children who lived 
longer distances from school. We also saw evidence from the qualitative and quantitative data 
that females tended to attend school more regularly than males. 


Although we found statistically significant effects on learning outcomes across the board, the 
results still suggested considerable scope for additional gains in learning outcomes. Despite the 
positive effects, children residing in IN catchment areas scored an average of only 10.9% on EGRA 


AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 60 


Midline Report for the Mixed-Methods Cluster-Randomized Controlled Trial of 
Impact Network’s eSchool 360 Model in Rural Zambia 


assessments and 24.3% on EGMA assessments. It will be critical to examine the longer-term 
effects of the eSchool 360 model to assess whether the program can achieve greater learning 
gains in future. 


Further, it will be important to investigate the cost-effectiveness of the eSchool 360 model 
using a rigorous cluster-RCT. Previous nonexperimental research has shown that the eSchool 
360 model may be cost-effective in achieving improvements in reading and mathematics 
outcomes. At present, however, it remains unclear whether a cluster-RCT will show the same 
results. A cluster-RCT of the outsourcing of education in Liberia showed that the program 
achieved large learning gains among some private providers, but program costs were 
considerable in some cases (Romero & Sandefur, 2019). Our cost-effectiveness analysis will 
examine whether the promise of the eSchool 360 model to achieve substantial learning gains 
for students in Zambian community schools for a moderate cost will hold true. 
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Appendix A. Bivariate Correlations With School Enrollment 


Table A-1. Demographic Characteristics and School Enrollment 


Not Enrolled Enrolled at 
FE mol aLole) | Ral store) | Difference Test 


p- 


Index child is female 0.27 0.03 | 0.00 0.54 


Index child is 8 years old or 051 0.50 0.01 | 0.03 | 0.71 -0.02 
older at baseline 
renee Seco 


Household size | 715 | 685 | 985 | 0.08 | 0.11 | 0.11 0.04 


CTs Elser O57) 715. | 067 || 673 | O41, aos 
school 


Highest level of education 
achieved by male household |! ae: || 743 ||-a34 |) ona.) 089 | B48 
member (or caregiver when 


jo) 


01 


oO 


.22 


jo) 


25 


there is no male adult) 


Katete district | 0.13 | 715 | 0.18 | 985 | 0.05 | 0.03 | 0.17 | 0.12 
Petauke district Fose | 715 | ose | 985 | -001 | 008 | 075 | -0.03 
Sinda district -0.07 
Index child ethnicity: Chewa | 0.37 | 709 | 0.38 | 978 | 0.01 | 0.04 | 0.75 | 0.03 
Index child ethnicity: Nsenga eto re ei el oe eel ae -0.04 
Index child ethnicity: Other ere eee ee 0.06 
Index child language: Chewa 0.34 34 0.06 


Nsenga 


Index child language: Other Gate paw ta 0.23 23 -0.07 
Index child is the 1st child oe oe ee 0.03 
Index child is the 2nd child | 0.38 | 666 | 0.41 | 807 | 0.03 | 0.00 | 0.05 | 0.07 
Index child is the 3rd child | 0.35 | 666 | 0.32 | 807 | -0.03 | 0.01 | 0.22 | -0.07 
Index child is the 4th child | 0.11 | 666 | 0.11 | 807 | -0.00 | 0.00 | 0.22 | -0.00 


Index child is the 5th child or 


0.03 | 666 | 0.01 807 -0.01 0.00 | 0.21 -0.08 
older 
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Note. Standard errors (SE) clustered at the school level. 


Table A-2. Household Socioeconomic Characteristics and School Enrollment 


Not Enrolled Enrolled at 
at School School Difference Test 


p- Std. Mean 
Variables N1 Value | Difference 


Household considers itself to be 
non-poor 
=| oon ow 


14.12, | 715 | 12.75 0.36 | 


Note. Standard errors (SE) clustered at the school level. 


Household considers itself to be 
moderately poor 


Household considers itself to be 
very poor 


Household food insecurity scale 


0.21 


Household asset index 


Table A-3. Baseline Learning Outcomes and School Enrollment 


Not Enrolled Enrolled at 
at School Aa sfore) | Difference Test 


p- Std. Mean 
Variables Value | Difference 


Zambian Achievement Test 0.01 | 00 0.25 
(percentage correct) 
Early Grade Reading 
Assessment (percentage 0.20 
correct) 
Early Grade Math Assessment 715 | 0.08 985 0.02 0.01 0.19 
(percentage correct) 
Oral Mocabularyipercentage: | 9.8. 94m |, Gen || 685 0.01 0.16 
correct) 
Standardized Zambian 

-0. ; : : 2 
Standardized Early Grade 

-0.1 71 11 2 : 0.25 
Standardized Early Grade Math 

-0.11 71 ; sal : oil, 
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Standardized Oral Vocabulary -0.09 | 715 | 0.07 985 0.16 0.05 | 0.00 0.16 


Note. Standard errors (SE) clustered at the school level. 


Appendix B. Adjusted p-Values for Multiple Comparisons in 
Heterogeneous ITT Effects 


Table B-1. Adjusted p-Values for Multiple Comparisons of Heterogenous Effects on Zambian 
Achievement Test (ZAT) (Standard Deviation) 


Standard Unadjusted Adjusted 
Sub-Groups Coefficient Error p-Value q-Value 


Treatment (ITT) X Female 0.023 0.090 0.802 0.959 
Treatment (ITT) X Index child was 8 or 9 0.113 0.109 0.302 0.959 
years old at baseline 

Treatment (ITT) X Caregiver attended -0.064 0.098 0.520 0.959 
school 

Treatment (ITT) X Index child scored -0.015 0.114 0.896 0.959 
above median ZAT at baseline 


Treatment (ITT) X Index child scored 0.007 0.129 0.959 0.959 
above median EGRA at baseline 

Sample with children who scored above 0.027 0.118 0.818 0.959 
median in EGMA at baseline 


Treatment (ITT) X Index child scored 0.159 0.103 0.126 0.959 
above median Oral Vocabulary at 
baseline 


Treatment (ITT) X Petauke 0.052 0.147 0.726 0.959 
Treatment (ITT) X Sinda 0.051 0.174 0.772 0.959 


Note. Standard errors clustered at the school level. 
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Table B-2. Adjusted p-Values for Multiple Comparisons of Heterogenous Effects on Zambian 
Achievement Test (ZAT) (Percentage Correct) 


Sit lirek-lae| Unadjusted Adjusted 
Sub-Groups Coefficient Error p-Value q-Value 


Treatment (ITT) X Female 0.004 0.018 0.802 0.959 
Treatment (ITT) X Index child was 8 or 9 0.022 0.021 0.302 0.959 
years old at baseline 

Treatment (ITT) X Caregiver attended -0.012 0.019 0.520 0.959 
school 

Treatment (ITT) X Index child scored -0.003 0.022 0.896 0.959 
above median ZAT at baseline 


Treatment (ITT) X Index child scored 0.001 0.025 0.959 0.959 
above median EGRA at baseline 

Sample with children who scored above 0.005 0.023 0.818 0.959 
median in EGMA at baseline 


Treatment (ITT) X Index child scored 0.031 0.020 0.126 0.959 
above median Oral Vocabulary at 
baseline 


Treatment (ITT) X Petauke 0.010 0.029 0.726 0.959 
Treatment (ITT) X Sinda 0.010 0.034 0.772 0.959 


Note. Standard errors clustered at the school level. 
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Table B-3. Adjusted p-Values for Multiple Comparisons of Heterogenous Effects on 
Standardized Early Grade Reading Assessment (EGRA) (Standard Deviation) 


Standard Unadjusted Adjusted 
Sub-Groups Coefficient Error p-Value q-Value 


Treatment (ITT) X Female 0.102 0.133 0.447 0.597 
Treatment (ITT) X Index child was 8 or 9 0.110 0.150 0.464 0.597 
years old at baseline 

Treatment (ITT) X Caregiver attended 0.070 0.134 0.604 0.604 
school 

Treatment (ITT) X Index child scored 0.132 0.105 0.214 0.432 
above median ZAT at baseline 


Treatment (ITT) X Index child scored 0.246 0.153 0.113 0.340 
above median EGRA at baseline 

Sample with children who scored above 0.338 0.131 0.012 0.063 
median in EGMA at baseline 


Treatment (ITT) X Index child scored 0.283 0.112 0.014 0.063 
above median Oral Vocabulary at 
baseline 


Treatment (ITT) X Petauke 0.141 0.258 0.586 0.604 
Treatment (ITT) X Sinda 0.295 0.248 0.240 0.432 


Note. Standard errors clustered at the school level. 
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Table B-4. Adjusted p-Values for Multiple Comparisons of Heterogenous Effects on Early 
Grade Reading Assessment (EGRA) (Percentage Correct) 


Sit lirek-lae| Unadjusted Adjusted 
Sub-Groups Coefficient Error p-Value q-Value 


Treatment (ITT) X Female 0.009 0.011 0.447 0.597 
Treatment (ITT) X Index child was 8 or 9 0.009 0.013 0.464 0.597 
years old at baseline 

Treatment (ITT) X Caregiver attended 0.006 0.012 0.604 0.604 
school 

Treatment (ITT) X Index child scored 0.011 0.009 0.214 0.432 
above median ZAT at baseline 


Treatment (ITT) X Index child scored 0.021 0.013 0.113 0.340 
above median EGRA at baseline 

Sample with children who scored above 0.029 0.011 0.012 0.063 
median in EGMA at baseline 


Treatment (ITT) X Index child scored 0.024 0.010 0.014 0.063 
above median Oral Vocabulary at 
baseline 


Treatment (ITT) X Petauke 0.012 0.022 0.586 0.604 
Treatment (ITT) X Sinda 0.025 0.021 0.240 0.432 


Note. Standard errors clustered at the school level. 
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Table B-5. Adjusted p-Values for Multiple Comparisons of Heterogenous Effects on 
Standardized Early Grade Mathematics Assessment (EGMA) (Standard Deviation) 


Sit lirek-lae| Unadjusted Adjusted 
Sub-Groups Coefficient Error p-Value q-Value 


Treatment (ITT) X Female -0.004 0.107 0.972 0.972 
Treatment (ITT) X Index child was 8 or 9 0.068 0.106 0.524 0.817 
years old at baseline 

Treatment (ITT) X Caregiver attended -0.059 0.123 0.633 0.817 
school 

Treatment (ITT) X Index child scored 0.103 0.100 0.305 0.817 
above median ZAT at baseline 


Treatment (ITT) X Index child scored 0.051 0.114 0.659 0.817 
above median EGRA at baseline 

Sample with children who scored above 0.061 0.122 0.621 0.817 
median in EGMA at baseline 


Treatment (ITT) X Index child scored 0.136 0.105 0.200 0.817 
above median Oral Vocabulary at 
baseline 


Treatment (ITT) X Petauke -0.216 0.197 0.277 0.817 
Treatment (ITT) X Sinda -0.069 0.197 0.726 0.817 


Note. Standard errors clustered at the school level. 
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Table B-6. Adjusted p-Values for Multiple Comparisons of Heterogenous Effects on Early 
Grade Mathematics Assessment (EGMA) (Percentage Correct) 


Sit lirek-lae| Unadjusted Adjusted 
Sub-Groups Coefficient Error p-Value q-Value 


Treatment (ITT) X Female -0.001 0.024 0.972 0.972 
Treatment (ITT) X Index child was 8 or 9 0.015 0.024 0.524 0.817 
years old at baseline 

Treatment (ITT) X Caregiver attended -0.013 0.028 0.633 0.817 
school 

Treatment (ITT) X Index child scored 0.023 0.022 0.305 0.817 
above median ZAT at baseline 


Treatment (ITT) X Index child scored 0.011 0.026 0.659 0.817 
above median EGRA at baseline 

Sample with children who scored above 0.014 0.027 0.621 0.817 
median in EGMA at baseline 


Treatment (ITT) X Index child scored 0.030 0.023 0.200 0.817 
above median Oral Vocabulary at 
baseline 


Treatment (ITT) X Petauke -0.048 0.044 0.277 0.817 
Treatment (ITT) X Sinda -0.016 0.044 0.726 0.817 


Note. Standard errors clustered at the school level. 
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Table B-7. Adjusted p-Values for Multiple Comparisons of Heterogenous Effects on 
Standardized Oral Vocabulary (Standard Deviation) 


Standard Unadjusted Adjusted 
Sub-Groups Coefficient Error PAV EV q-Value 


Treatment (ITT) X Female -0.116 0.096 0.231 0.790 
Treatment (ITT) X Index child was 8 or 9 0.102 0.093 0.273 0.790 
years old at baseline 

Treatment (ITT) X Caregiver attended -0.037 0.095 0.696 0.845 
school 

Treatment (ITT) X Index child scored 0.023 0.103 0.824 0.845 
above median ZAT at baseline 


Treatment (ITT) X Index child scored 0.075 0.118 0.527 0.790 
above median EGRA at baseline 

Sample with children who scored above 0.078 0.101 0.441 0.790 
median in EGMA at baseline 


Treatment (ITT) X Index child scored 0.020 0.102 0.845 0.845 
above median Oral Vocabulary at 
baseline 


Treatment (ITT) X Petauke -0.136 0.165 0.411 0.790 
Treatment (ITT) X Sinda -0.200 0.182 0.277 0.790 


Note. Standard errors clustered at the school level. 
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Table B-8. Adjusted p-Values for Multiple Comparisons of Heterogenous Effects on Oral 
Vocabulary (Percentage Correct) 


Sitlirek-lae| Unadjusted Adjusted 
Sub-Groups Coefficient Error p-Value q-Value 


Treatment (ITT) X Female -0.028 0.023 0.231 0.790 
Treatment (ITT) X Index child was 8 or 9 0.025 0.022 0.273 0.790 
years old at baseline 

Treatment (ITT) X Caregiver attended -0.009 0.023 0.696 0.845 
school 

Treatment (ITT) X Index child scored 0.006 0.025 0.824 0.845 
above median ZAT at baseline 


Treatment (ITT) X Index child scored 0.018 0.028 0.527 0.790 
above median EGRA at baseline 

Sample with children who scored above 0.019 0.024 0.441 0.790 
median in EGMA at baseline 


Treatment (ITT) X Index child scored 0.005 0.025 0.845 0.845 
above median Oral Vocabulary at 
baseline 


Treatment (ITT) X Petauke -0.033 0.040 0.411 0.790 
Treatment (ITT) X Sinda -0.048 0.044 0.277 0.790 


Note. Standard errors clustered at the school level. 
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Appendix C. Correlations between Intermediate and Learning 
Outcomes 


Table C-1. Correlations Between Intermediate and Learning Outcomes for Complete Sample 


Se ae eee 
Oral Vocabulary 


ZAT (SMD) EGRA (SMD) | EGMA (SMD) (SMD) 


Index child was enrolled at any 0.376*** 0.257* 0.368*** 0.289** 
school (last year or this year) (0.089) (0.101) (0.079) (0.091) 
school last week (0.019) (0.030) (0.016) (0.019) 

0.061*** 0.059*** 0.065*** 0.053*** 
Child development scale 

(0.014) (0.010) (0.009) (0.012) 

-0.039 0.223 0.164 0.273* 
Petauke 

(0.084) (0.138) (0.104) (0.120) 

0.082 0.269+ 0.150 0.327** 
Sinda 

= 078) ae 142) are 092) aa 111) 


Highest level of education | 0.003 | 003 | 0.013, 013 | 0.008 | 008 | 0.002 | 002 


achieved by male household (0.007) (0.012) (0.007) (0.007) 
member 

0. | o.os3* | 0.127** OMA as 0. | oose*# 
Index child age 

(0.028) (0.041) (0.027) (0.027) 

0.024 0.056 0.019 0.069 

Index child is a female 

(0.051) (0.065) (0.045) (0.057) 

-0.156* -0.066 -0.192** -0.224*** 
Index child is a biological child 

(0.063) (0.079) (0.069) (0.055) 


0.015 -0.001 -0.007 0.006 


Household size 
(0.013) (0.018) (0.014) (0.015) 


0.079 0.106+ 0.103+ 0.076 


Caregiver has attended school 
(0.052) (0.060) (0.055) (0.050) 


0.443 -0.077 0.028 0.161 


Index child ethnicity: Tumbuka 
(0.462) (0.225) (0.196) 


0.074 -0.210 0.109 -0.274 


Index child language: Chewa 
(0.193) (0.563) (0.318) (0.293) 
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Index child language: Nsenga 


Index child is the 1st child 


Index child is the 2nd child 


Index child is the 3rd child 


Index child is the 4th child 


Household food insecurity 
access scale 


Household asset index 


Household distance from 
school (km) 


Baseline standardized ZAT 
score 


Baseline standardized EGRA 
score 


Baseline standardized EGMA 
score 


Baseline standardized Oral 
Vocabulary score 


Constant 
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a a ee 
Oral Vocabulary 


ZAT (SMD) EGRA (SMD) | EGMA (SMD) CHVT)) 


a 
Fe 
Se 
Se 
Se 
Se 
ee 
ee 


Adjusted R? 0.220 0.170 0.266 0.143 


Note. Standard errors in parentheses. + p < 0.10, * p< 0.05, ** p< 0.01, *** p< 0.001 
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Table C-2. Correlations Between Intermediate and Learning Outcomes for the Sample 


6) 04) (3) 
ZAT (SMD) Oral Vocabulary 
IV Te)) 
Number of days attended 0.050* 0.125** 0.058* 0.016 
school last week (0.023) (0.045) (0.024) (0.022) 
0.033* 0.062** 0.053** 0.022+ 
Child development scale 
(0.015) (0.022) (0.019) (0.011) 
Age at which index child first -0.600*** -0.986*** -0.845*** -0.397*** 
enrolled at school (0.065) (0.103) (0.055) (0.064) 
Caregiver 0.021 0.007 0.014 0.013 
perception/engagement scale (0.013) (0.020) (0.013) (0.013) 
0.091** 0.279*** 0.108** 0.059* 
Caregiver satisfaction scale 
(0.027) (0.059) (0.034) (0.029) 
Highest education level of male “0.002 0.022 “0.005 0.008 
household member (0.010) (0.017) (0.009) (0.007) 
-0.068 0.173 0.042 0.068 
Petauke 
(0.120) (0.227) (0.139) (0.115) 
0.040 0.211 0.021 0.128 
Sinda 
(0.114) (0.246) (0.134) (0.110) 
0.724*** 1.184*** 1.050*** 0.484*** 
Index child age 
(0.071) (0.145) (0.067) (0.063) 
-0.025 0.024 -0.026 0.065 
Index child is a female 
(0.067) (0.108) (0.064) (0.072) 
-0.235** -0.092 -0.190+ -0.220*** 
Index child is a biological child 
(0.082) (0.123) (0.102) (0.059) 
0.017 -0.038 -0.031 0.013 
Household size 
(0.017) (0.030) (0.020) (0.016) 
0.087 0.148+ 0.137+ 0.084 
Caregiver has attended school 
(0.058) (0.083) (0.075) (0.065) 
Index child ethnicity: Tumbuka -0.144 -0.360 -0.381 0.143 


Enrolled in School 
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Index child language: Chewa 


Index child language: Nsenga 


Index child is the 1st child 


Index child is the 2nd child 


Index child is the 3rd child 


Index child is the 4th child 


Household food insecurity 
access scale 


Household asset index 


Baseline household distance 
from school (km) 


Baseline standardized EGRA 
score 


Baseline standardized EGMA 
score 


Baseline standardized Oral 
Vocabulary score 


Constant 


Baseline standardized ZAT score 
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ZAT (SMD) 


(0.313) 
0.014 
(0.323) 
-0.030 
(0.323) 
0.159 
(0.218) 
0.098 
(0.197) 
0.130 


0.196 


om | ao 


-0.010+ 


-0.700 
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EGRA (SMD) | EGMA (SMD) 


-0.989 -0.186 
(1.145) (0.630) 
-1.261 -0.133 
0.076 -0.121 
(0.369) (0.307) 
0.168 -0.059 
0.067 -0.033 
0.673 0.146 


(0.440) 


0.065+ 
a 
a 
0.033 Se 
(0.052) | | 
0.156*** 


(0.037) 


-0.481 -0.880 


(1.229) (0.743) 


Oral Vocabulary 


(CLY1>)) 
(0.203) 
-0.383 
(0.477) 
-0.278 
(0.479) 
0.121 

(0.196) 
-0.028 
(0.186) 
0.026 

(0.193) 
0.220 

(0.192) 

-0.016** 

(0.005) 
0.012 

(0.032) 
0.048 

(0.036) 


0.083* 
(0.033) 
-0.129 
(0.659) 
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ZAT (SMD) EGRA (SMD) | EGMA(SMD) | Oral Vocabulary 


(SMD) 


Adjusted R? 0.168 0.171 0.242 0.123 


Note. Standard errors in parentheses. + p < 0.10, * p< 0.05, ** p<0.01, *** p< 0.001 
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Appendix D. ITT Effects on Probability of Non-Zero Scores 


Table D-1. ITT Effects on Probability of Scoring Above Zero 


VARIABLES Oral Vocabulary 


0.006 0.047** 0.017** 0.022** 
Treatment 


(0.007) (0.019) (0.008) (0.009) 
0.006 | 006 0.012 oon 


Control group mean 0.981 0.834 0.963 0.967 


Note. Standard errors in parentheses. * p< 0.10, ** p<0.05, *** p<0.01 
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